一般操作:
-----
hbase(main)> status
hbase(main)> version
创建命名空间: namespace指的是一个 表的逻辑分组 ,同一组中的表有类似的用途,相当于关系型数据库中的database。
hbase(main):060:0> create_namespace 'test1'
drop_namespace
创建该命名空间的表:
hbase(main):061:0> create 'test1:test','f1','f2'
create 'scores',{NAME=>'course',VERSIONS=>2}
1)查看有哪些表 list describe
hbase(main)> list
hbase(main)> describe 'member'
2)创建表 create 只创建列族就可,put数据时直接
# 语法:create <table>, {NAME => <family>, VERSIONS => <VERSIONS>}
# 例如:创建表t1,有两个family name:f1,f2,且版本数均为2
hbase(main)> create 't1',{NAME => 'f1', VERSIONS => 2},{NAME => 'f2', VERSIONS => 2}
3)删除表
分两步:首先disable,然后drop
例如:删除表t1
hbase(main)> disable 't1'
hbase(main)> drop 't1'
4)查看表的结构
# 语法:describe <table>
# 例如:查看表t1的结构
hbase(main)> describe 't1'
5)修改表结构 alter
修改表结构必须先disable
alter 't1', {NAME => 'f1', VERSIONS => 5}
# 语法:alter 't1', {NAME => 'f1'}, {NAME => 'f2', METHOD => 'delete'}
# 例如:修改表test1的cf的TTL为180天
hbase(main)> disable 'test1'
hbase(main)> alter 'test1',{NAME=>'body',TTL=>'15552000'},{NAME=>'meta', TTL=>'15552000'}
hbase(main)> enable 'test1'
6)添加数据 put
# 语法:put <table>,<rowkey>,<family:column>,<value>,<timestamp>
# 例如:给表t1的添加一行记录:rowkey是rowkey001,family name:f1,column name:col1,value:value01,timestamp:系统默认
hbase(main)> put 't1','rowkey001','f1:col1','value01'
用法比较单一。
7)查询数据
a)查询某行记录 get
# 语法:get <table>,<rowkey>,[<family:column>,....]
查询rowkey001一行下的所有列值:
hbase(main)> get 't1','rowkey001'
# 例如:查询表t1,rowkey001行,f1:col1列
hbase(main)> get 't1','rowkey001', 'f1:col1'
# 或者:
hbase(main)> get 't1','rowkey001', {COLUMN=>'f1:col1'}
b)扫描表
# 语法:scan <table>, {COLUMNS => [ <family:column>,.... ], LIMIT => num}
# 另外,还可以添加STARTROW、TIMERANGE和FITLER等高级功能
# 例如:扫描表t1的前5条数据
hbase(main)> scan 't1',{LIMIT=>5}
test1空间下的test表的columns=f1的所有行
hbase(main)> scan 'test1:test',{COLUMNS=>'f1'}
test1空间下的test表的columns=f1的第1行
hbase(main)> scan 'test1:test',{COLUMNS=>'f1',LIMIT=>1}
scan 'scores',{VERSIONS=>2} version<=2
scan 'scores',{TIMERANGE=>[1394097631386,1394097651029],VERSIONS=>2}
c)查询表中的数据行数 count
# 语法:count <table>, {INTERVAL => intervalNum, CACHE => cacheNum}
# INTERVAL设置多少行显示一次及对应的rowkey,默认1000;CACHE每次去取的缓存区大小,默认是10,调整该参数可提高查询速度
# 例如,查询表t1中的行数,每100条显示一次,缓存区为500
hbase(main)> count 't1', {INTERVAL => 100, CACHE => 500}
8)删除数据
a )删除行中的某个列值 delete
# 语法:delete <table>, <rowkey>, <family:column> , <timestamp>,必须指定列名
# 例如:删除表t1,rowkey001中的f1:col1的数据
hbase(main)> delete 't1','rowkey001','f1:col1'
注:将删除改行f1:col1列所有版本的数据
b )删除行 deleteall
# 语法:deleteall <table>, <rowkey>, <family:column> , <timestamp>,可以不指定列名,删除整行数据
# 例如:删除表t1,rowk001的数据
hbase(main)> deleteall 't1','rowkey001'
c)删除表中的所有数据 truncate
# 语法: truncate <table>
# 其具体过程是:disable table -> drop table -> create table
# 例如:删除表t1的所有数据
hbase(main)> truncate 't1'
9)检查表是否存在 exists
hbase(main):019:0> exists 't1'
10)查看表是否可用 is_enabled
hbase(main):036:0> is_enabled 't1'
help
hbase(main)> create help
11).判断表是否enable
hbase(main):034:0>is_enabled 'member'
**过滤器:**
--------
0.所有的过滤器都在服务端生效
1.显示所有过滤器
hbase(main):010:0> show_filters
2.只返回key及其他关键部分
scan 'airline',{ FILTER => "KeyOnlyFilter()"}
3.只返回每一行的第一个值
scan 'airline',{ FILTER => "FirstKeyOnlyFilter()"}
4.过滤rowkey 需要输入rowkey的前缀
scan 'airline', {FILTER => "(PrefixFilter ('row2'))"}
5.过虑qualifier,需要输入qualifier前缀
scan 'airline', {FILTER => "(PrefixFilter ('row2')) AND ColumnPrefixFilter('destination')"}
6.多重过滤qualifier,需要输入qualifier前缀
scan 'airline',{FILTER =>"MultipleColumnPrefixFilter('source','destination','date')"}
7.返回该表中相应的qualifier数
scan 'airline',{FILTER =>"ColumnCountGetFilter(2)"}
8.返回多少行
scan 'airline',{FILTER => "PageFilter(1)"}
9.扫描到哪一行停止
scan 'airline',{FILTER =>"InclusiveStopFilter('row1')"}
10.只返回有指定Qualifier的数据
scan 'airline',{ FILTER =>"QualifierFilter(=,'binary:flightno')"} > = <来替换‘=’
11.返回满足条件(某一列的值)的数据
scan 'airline', { COLUMNS =>'flightbetween:source', LIMIT => 4, FILTER => "ValueFilter( =, 'binaryprefix:hyd' )" } > = <来替换‘=’
行的最大版本是通过 HColumnDescriptor定义在每一个列族的,默认的最大版本号是1
不推荐设置 最大版本号 为很大的值(数百甚至更多),除非旧的数据对你而言十分重要。因为太多的版本会使 StoreFile 很大。
hbase org.apache.hadoop.hbase.mapreduce.Driver import apply_info /user/data_temp/apply_info hdfs路径 hbase表
建表语句详解:
-------
create 'testtable',{NAME => 'Toutiao', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '10', COMPRESSION => 'LZO', TTL => '30000', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
{NAME => 'coulmn', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '30', COMPRESSION => 'LZO', TTL => '30000', IN_MEMORY => 'true'}
(其中的属性有 REPLICATION_SCOPE 复制份数 ,只包含一个列簇“Toutiao”,versions:设置历史版本数 默认为1,TTL:过期时间 单位为秒,默认为永久保存,COMPRESSION:压缩方式,当配置lzo的情况)
BLOOMFILTER
布隆过滤器,优化HBase的随即读取性能,可选值NONE|ROW|ROWCOL,默认为NONE,该参数可以单独对某个列簇启用。启用过滤器,对于get操作以及部分scan操作可以剔除掉不会用到的存储文件,减少实际IO次数,提高随机读性能。Row类型适用于只根据Row进行查找,而RowCol类型适用于根据Row+Col联合查找,如下:
Row类型适用于:get ‘NewsClickFeedback’,’row1′
RowCol类型适用于:get ‘NewsClickFeedback’,’row1′,{COLUMN => ‘Toutiao’}
对于有随机读的业务,建议开启Row类型的过滤器,使用空间换时间,提高随机读性能。
COMPRESSION
数据压缩方式,HBase支持多种形式的数据压缩,一方面减少数据存储空间,一方面降低数据网络传输量进而提升读取效率。目前HBase支持的压缩算法主要包括三种:GZip | LZO | Snappy,下面表格分别从压缩率,编解码速率三个方面对其进行对比:
Snappy的压缩率最低,但是编解码速率最高,对CPU的消耗也最小,目前一般建议使用Snappy
IN_MEMORY
数据是否常驻内存,默认为false。HBase为频繁访问的数据提供了一个缓存区域,缓存区域一般存储数据量小、访问频繁的数据,常见场景为元数据存储。默认情况,该缓存区域大小等于Jvm Heapsize * 0.2 * 0.25 ,假如Jvm Heapsize = 70G,存储区域的大小约等于3.2G。需要注意的是HBase Meta元数据信息存储在这块区域,如果业务数据设置为true而且太大会导致Meta数据被置换出去,导致整个集群性能降低,所以在设置该参数时需要格外小心。
BLOCKCACHE
是否开启block cache缓存,默认开启。
TTL
数据过期时间,单位为秒,默认为永久保存。对于很多业务来说,有时候并不需要永久保存某些数据,永久保存会导致数据量越来越大,消耗存储空间是其一,另一方面还会导致查询效率降低。如果设置了过期时间
HBase过滤器简介
(1) 过滤器简介
过滤器 | 解释 |
ColumnPrefixFilter | 列前缀过滤器 |
TimestampsFilter | 时间戳过滤器 |
PageFilter | 分页过滤器 |
MultipleColumnPrefixFilter | 复合列前缀过滤器 |
FamilyFilter | 列簇过滤器 |
ColumnPaginationFilter | |
SingleColumnValueFilter | 单列值过滤器 |
RowFilter | 行健过滤器 |
QualifierFilter | 列过滤器 |
ColumnRangeFilter | |
ValueFilter | 值过滤器 |
PrefixFilter | 前缀过滤器 |
SingleColumnValueExcludeFilter | 单列值排除器 |
ColumnCountGetFilter | |
InclusiveStopFilter | |
DependentColumnFilter | |
FirstKeyOnlyFilter | |
KeyOnlyFilter |
(2) 过滤器分类
类别 | 过滤器 |
比较过滤器 | RowFilter、FamilyFilter、QualifierFilter、ValueFilter 等 |
专用过滤器 | SingleColumnValueFilter、SingleColumnValueExcludeFilter、PrefixFilter、ColumnPrefixFilter、PageFilter 等 |
(3) 运算符种类
运算符 | 说明 |
LESS | < |
LESS_OR_EQUAL | <= |
EQUAL | = |
NOT_EQUAL | != |
GREATER_OR_EQUAL | >= |
GREATER | > |
NO_OP | 没有运算符 |
(4) 比较器种类
比较器 | 说明 |
BinaryComparator | 按字节索引顺序比较指定字节数组,采用 Bytes.compareTo(byte[]) |
BinaryPrefixComparator | 跟前面相同,只是比较左端的数据是否相同 |
BitComparator | 按位比较 |
LongComparator | 比较long型value |
NullComparator | 判断给定value的是否为空 |
RegexStringComparator | 提供一个正则的比较器,仅支持 EQUAL 和 NOT_EQUAL 运算符 |
SubstringComparator | 判断提供的子串是否出现在 value 中 |
(5) 使用比较过滤器的方法
比较运算符 CompareFilter.CompareOp 比较运算符用于定义比较关系,可以有以下几类值供选择:
- EQUAL 相等
- GREATER 大于
- GREATER_OR_EQUAL 大于等于
- LESS 小于
- LESS_OR_EQUAL 小于等于
- NOT_EQUAL 不等于
比较器 ByteArrayComparable 通过比较器可以实现多样化目标匹配效果,比较器有以下子类可以使用:
- BinaryComparator 匹配完整字节数组
- BinaryPrefixComparator 匹配字节数组前缀
- BitComparator 不常用
- NullComparator 不常用
- RegexStringComparator 匹配正则表达式
- SubstringComparator 匹配子字符串
1.多重过滤器--FilterList(Shell不支持) FilterList代表一个过滤器链,它可以包含一组即将应用于目标数据集的过滤器,过滤器间具有“与” FilterList.Operator.MUST_PASS_ALL 和“或” FilterList.Operator.MUST_PASS_ONE 关系。
//结合过滤器,获取所有age在15到30之间的行
private static void scanFilter() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
// And
FilterList filterList = new FilterList(FilterList.Operator.MUST_PASS_ALL);
// >=15
SingleColumnValueFilter filter1 = new SingleColumnValueFilter("info".getBytes(), "age".getBytes(), CompareOp.GREATER_OR_EQUAL, "15".getBytes());
// =<30
SingleColumnValueFilter filter2 = new SingleColumnValueFilter("info".getBytes(), "age".getBytes(), CompareOp.LESS_OR_EQUAL, "30".getBytes());
filterList.addFilter(filter1);
filterList.addFilter(filter2);
Scan scan = new Scan();
// set Filter
scan.setFilter(filterList);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
2. 列值过滤器--SingleColumnValueFilter 用于测试列值相等(CompareOp.EQUAL ),不等(CompareOp.NOT_EQUAL),或单侧范围 (如CompareOp.GREATER)。构造函数: 2.1.比较的关键字是一个字符数组(Shell不支持?) SingleColumnValueFilter(byte[] family, byte[] qualifier, CompareFilter.CompareOp compareOp, byte[] value)
//SingleColumnValueFilter例子
private static void scanFilter01() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
SingleColumnValueFilter scvf = new SingleColumnValueFilter("info".getBytes(), "age".getBytes(), CompareOp.EQUAL, "18".getBytes());
Scan scan = new Scan();
scan.setFilter(scvf);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
2.2.比较的关键字是一个比较器ByteArrayComparable SingleColumnValueFilter(byte[] family, byte[] qualifier, CompareFilter.CompareOp compareOp, ByteArrayComparable comparator)
hbase(main):032:0> scan 'users',{FILTER=>"SingleColumnValueFilter('info','age',=,'regexstring:.4')"}
ROW COLUMN+CELL
xiaoming01 column=address:contry, timestamp=1442000277200, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=address:country, timestamp=1442000228945, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=info:age, timestamp=1441998917568, value=24
xiaoming02 column=info:age, timestamp=1441998917594, value=24
xiaoming03 column=info:age, timestamp=1441998919607, value=24
3 row(s) in 0.0130 seconds
//SingleColumnValueFilter例子2 -- RegexStringComparator
private static void scanFilter02() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//值比较的正则表达式 -- RegexStringComparator
//匹配info:age值以"4"结尾
RegexStringComparator comparator = new RegexStringComparator(".4");
//第四个参数不一样
SingleColumnValueFilter scvf = new SingleColumnValueFilter("info".getBytes(), "age".getBytes(), CompareOp.EQUAL, comparator);
Scan scan = new Scan();
scan.setFilter(scvf);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
//SingleColumnValueFilter例子2 -- SubstringComparator
private static void scanFilter03() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//检测一个子串是否存在于值中(大小写不敏感) -- SubstringComparator
//过滤age值中包含'4'的RowKey
SubstringComparator comparator = new SubstringComparator("4");
//第四个参数不一样
SingleColumnValueFilter scvf = new SingleColumnValueFilter("info".getBytes(), "age".getBytes(), CompareOp.EQUAL, comparator);
Scan scan = new Scan();
scan.setFilter(scvf);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
hbase(main):033:0> scan 'users',{FILTER=>"SingleColumnValueFilter('info','age',=,'substring:4')"}
ROW COLUMN+CELL
xiaoming01 column=address:contry, timestamp=1442000277200, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=address:country, timestamp=1442000228945, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=info:age, timestamp=1441998917568, value=24
xiaoming02 column=info:age, timestamp=1441998917594, value=24
xiaoming03 column=info:age, timestamp=1441998919607, value=24
3 row(s) in 0.0180 seconds
3.列名过滤器 由于HBase采用键值对保存内部数据,列名过滤器过滤一行的列名(ColumnFamily:Qualifiers)是否存在 , 对应前节所述列值的情况。
3.1.基于Columun Family列族过滤数据的FamilyFilter FamilyFilter(CompareFilter.CompareOp familyCompareOp, ByteArrayComparable familyComparator)
注意: 1.如果希望查找的是一个已知的列族,则使用 scan.addFamily(family); 比使用过滤器效率更高. 2.由于目前HBase对多列族支持不完善,所以该过滤器目前用途不大.
//基于列族过滤数据的FamilyFilter
private static void scanFilter04() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//过滤 = 'address'的列族
//FamilyFilter familyFilter = new FamilyFilter(CompareOp.EQUAL, new BinaryComparator("address".getBytes()));
//过滤以'add'开头的列族
FamilyFilter familyFilter = new FamilyFilter(CompareOp.EQUAL, new BinaryPrefixComparator("add".getBytes()));
Scan scan = new Scan();
scan.setFilter(familyFilter);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
hbase(main):021:0> scan 'users',{FILTER=>"FamilyFilter(=,'binaryprefix:add')"}
ROW COLUMN+CELL
xiaoming column=address:city, timestamp=1441997498965, value=hangzhou
xiaoming column=address:contry, timestamp=1441997498911, value=china
xiaoming column=address:province, timestamp=1441997498939, value=zhejiang
xiaoming01 column=address:contry, timestamp=1442000277200, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=address:country, timestamp=1442000228945, value=\xE4\xB8\xAD\xE5\x9B\xBD
zhangyifei column=address:city, timestamp=1441997499108, value=jieyang
zhangyifei column=address:contry, timestamp=1441997499077, value=china
zhangyifei column=address:province, timestamp=1441997499093, value=guangdong
zhangyifei column=address:town, timestamp=1441997500711, value=xianqiao
3 row(s) in 0.0400 seconds
3.2.基于Qualifier列名过滤数据的QualifierFilter QualifierFilter(CompareFilter.CompareOp op, ByteArrayComparable qualifierComparator)
说明:该过滤器应该比FamilyFilter更常用!
//基于Qualifier(列名)过滤数据的QualifierFilter
private static void scanFilter05() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//过滤列名 = 'age'所有RowKey
//QualifierFilter qualifierFilter = new QualifierFilter(CompareOp.EQUAL, new BinaryComparator("age".getBytes()));
//过滤列名 以'age'开头 所有RowKey(包含age)
//QualifierFilter qualifierFilter = new QualifierFilter(CompareOp.EQUAL, new BinaryPrefixComparator("age".getBytes()));
//过滤列名 包含'age' 所有RowKey(包含age)
//QualifierFilter qualifierFilter = new QualifierFilter(CompareOp.EQUAL, new SubstringComparator("age"));
//过滤列名 符合'.ge'正则表达式 所有RowKey
QualifierFilter qualifierFilter = new QualifierFilter(CompareOp.EQUAL, new RegexStringComparator(".ge"));
Scan scan = new Scan();
scan.setFilter(qualifierFilter);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
hbase(main):020:0> scan 'users',{FILTER=>"QualifierFilter(=,'regexstring:.ge')"}
ROW COLUMN+CELL
xiaoming column=info:age, timestamp=1441997971945, value=38
xiaoming01 column=info:age, timestamp=1441998917568, value=24
xiaoming02 column=info:age, timestamp=1441998917594, value=24
xiaoming03 column=info:age, timestamp=1441998919607, value=24
zhangyifei column=info:age, timestamp=1442247255446, value=18
5 row(s) in 0.0460 seconds
3.3.基于列名前缀过滤数据的ColumnPrefixFilter(该功能用QualifierFilter也能实现) ColumnPrefixFilter(byte[] prefix) 注意:一个列名是可以出现在多个列族中的,该过滤器将返回所有列族中匹配的列。
//ColumnPrefixFilter例子
private static void scanFilter06() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//匹配 以'ag'开头的所有的列
ColumnPrefixFilter columnPrefixFilter = new ColumnPrefixFilter("ag".getBytes());
Scan scan = new Scan();
scan.setFilter(columnPrefixFilter);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
hbase(main):018:0> scan 'users',{FILTER=>"ColumnPrefixFilter('ag')"}
ROW COLUMN+CELL
xiaoming column=info:age, timestamp=1441997971945, value=38
xiaoming01 column=info:age, timestamp=1441998917568, value=24
xiaoming02 column=info:age, timestamp=1441998917594, value=24
xiaoming03 column=info:age, timestamp=1441998919607, value=24
zhangyifei column=info:age, timestamp=1442247255446, value=18
5 row(s) in 0.0280 seconds
3.4.基于多个列名前缀过滤数据的MultipleColumnPrefixFilter MultipleColumnPrefixFilter 和 ColumnPrefixFilter 行为差不多,但可以指定多个前缀。
//MultipleColumnPrefixFilter例子
private static void scanFilter07() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//匹配 以'a'或者'c'开头 所有的列{二维数组}
byte[][] prefixes =new byte[][]{"a".getBytes(), "c".getBytes()};
MultipleColumnPrefixFilter multipleColumnPrefixFilter = new MultipleColumnPrefixFilter(prefixes );
Scan scan = new Scan();
scan.setFilter(multipleColumnPrefixFilter);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
hbase(main):017:0> scan 'users',{FILTER=>"MultipleColumnPrefixFilter('a','c')"}
ROW COLUMN+CELL
xiaoming column=address:city, timestamp=1441997498965, value=hangzhou
xiaoming column=address:contry, timestamp=1441997498911, value=china
xiaoming column=info:age, timestamp=1441997971945, value=38
xiaoming column=info:company, timestamp=1441997498889, value=alibaba
xiaoming01 column=address:contry, timestamp=1442000277200, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=address:country, timestamp=1442000228945, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=info:age, timestamp=1441998917568, value=24
xiaoming02 column=info:age, timestamp=1441998917594, value=24
xiaoming03 column=info:age, timestamp=1441998919607, value=24
zhangyifei column=address:city, timestamp=1441997499108, value=jieyang
zhangyifei column=address:contry, timestamp=1441997499077, value=china
zhangyifei column=info:age, timestamp=1442247255446, value=18
zhangyifei column=info:company, timestamp=1441997499039, value=alibaba
5 row(s) in 0.0430 seconds
3.5.基于列范围(不是行范围)过滤数据ColumnRangeFilter
- 可用于获得一个范围的列,例如,如果你的一行中有百万个列,但是你只希望查看列名从bbbb到dddd的范围
- 该方法从 HBase 0.92 版本开始引入
- 一个列名是可以出现在多个列族中的,该过滤器将返回所有列族中匹配的列
构造函数: ColumnRangeFilter(byte[] minColumn, boolean minColumnInclusive, byte[] maxColumn, boolean maxColumnInclusive) 参数解释:
- minColumn - 列范围的最小值,如果为空,则没有下限
- minColumnInclusive - 列范围是否包含minColumn
- maxColumn - 列范围最大值,如果为空,则没有上限
- maxColumnInclusive - 列范围是否包含maxColumn
//ColumnRangeFilter例子
private static void scanFilter08() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//匹配 以'a'开头到以'c'开头(不包含c) 所有的列
ColumnRangeFilter columnRangeFilter = new ColumnRangeFilter("a".getBytes(), true, "c".getBytes(), false);
Scan scan = new Scan();
scan.setFilter(columnRangeFilter);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
hbase(main):016:0> scan 'users',{FILTER=>"ColumnRangeFilter('a',true,'c',false)"}
ROW COLUMN+CELL
xiaoming column=info:age, timestamp=1441997971945, value=38
xiaoming column=info:birthday, timestamp=1441997498851, value=1987-06-17
xiaoming01 column=info:age, timestamp=1441998917568, value=24
xiaoming02 column=info:age, timestamp=1441998917594, value=24
xiaoming03 column=info:age, timestamp=1441998919607, value=24
zhangyifei column=info:age, timestamp=1442247255446, value=18
zhangyifei column=info:birthday, timestamp=1441997498990, value=1987-4-17
5 row(s) in 0.0340 seconds
4.RowKey 当需要根据行键特征查找一个范围的行数据时,使用Scan的startRow和stopRow会更高效,但是,startRow和stopRow只能匹配行键的开始字符,而不能匹配中间包含的字符。当需要针对行键进行更复杂的过滤时,可以使用RowFilter。 构造函数:RowFilter(CompareFilter.CompareOp rowCompareOp, ByteArrayComparable rowComparator)
//RowFilter例子
private static void scanFilter09() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//匹配 行键包含'01' 所有的行
RowFilter rowFilter = new RowFilter(CompareOp.EQUAL, new SubstringComparator("01"));
Scan scan = new Scan();
scan.setFilter(rowFilter);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
hbase(main):013:0> scan 'users',{FILTER=>"RowFilter(=,'substring:01')"}
ROW COLUMN+CELL
xiaoming01 column=address:contry, timestamp=1442000277200, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=address:country, timestamp=1442000228945, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming01 column=info:age, timestamp=1441998917568, value=24
1 row(s) in 0.0190 seconds
5.PageFilter(Shell不支持?) 指定页面行数,返回对应行数的结果集。 需要注意的是,该过滤器并不能保证返回的结果行数小于等于指定的页面行数,因为过滤器是分别作用到各个region server的,它只能保证当前region返回的结果行数不超过指定页面行数。 构造函数:PageFilter(long pageSize)
//PageFilter例子
private static void scanFilter10() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//从RowKey为 "xiaoming" 开始,取3行(包含xiaoming)
PageFilter pageFilter = new PageFilter(3L);
Scan scan = new Scan();
scan.setStartRow("xiaoming".getBytes());
scan.setFilter(pageFilter);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
注意:由于该过滤器并不能保证返回的结果行数小于等于指定的页面行数,所以更好的返回指定行数的办法是ResultScanner.next(int nbRows),即:
//上面Demo的改动版
private static void scanFilter11() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//从RowKey为 "xiaoming" 开始,取3行(包含xiaoming)
//PageFilter pageFilter = new PageFilter(3L);
Scan scan = new Scan();
scan.setStartRow("xiaoming".getBytes());
//scan.setFilter(pageFilter);
ResultScanner rs = ht.getScanner(scan);
//指定返回3行数据
for(Result result : rs.next(3)){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
6.SkipFilter(Shell不支持) 根据整行中的每个列来做过滤,只要存在一列不满足条件,整行都被过滤掉。 构造函数:SkipFilter(Filter filter)
例如,如果一行中的所有列代表的是不同物品的重量,则真实场景下这些数值都必须大于零,我们希望将那些包含任意列值为0的行都过滤掉。在这个情况下,我们结合ValueFilter和SkipFilter共同实现该目的: scan.setFilter(new SkipFilter(new ValueFilter(CompareOp.NOT_EQUAL,new BinaryComparator(Bytes.toBytes(0))));
//SkipFilter例子
private static void scanFilter12() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//跳过列值中包含"24"的所有列
SkipFilter skipFilter = new SkipFilter(new ValueFilter(CompareOp.NOT_EQUAL, new BinaryComparator("24".getBytes())));
Scan scan = new Scan();
scan.setFilter(skipFilter);
ResultScanner rs = ht.getScanner(scan);
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
}
}
ht.close();
}
7.Utility--FirstKeyOnlyFilter 该过滤器仅仅返回每一行中第一个cell的值,可以用于高效的执行行数统计操作。估计实战意义不大。 构造函数:public FirstKeyOnlyFilter()
//FirstKeyOnlyFilter例子
private static void scanFilter12() throws IOException,
UnsupportedEncodingException {
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.rootdir", "hdfs://ncst:9000/hbase");
conf.set("hbase.zookeeper.quorum", "ncst");
HTable ht = new HTable(conf, "users");
//返回每一行中的第一个cell的值
FirstKeyOnlyFilter firstKeyOnlyFilter = new FirstKeyOnlyFilter();
Scan scan = new Scan();
scan.setFilter(firstKeyOnlyFilter);
ResultScanner rs = ht.getScanner(scan);
int i = 0;
for(Result result : rs){
for(Cell cell : result.rawCells()){
System.out.println(new String(CellUtil.cloneRow(cell))+"\t"
+new String(CellUtil.cloneFamily(cell))+"\t"
+new String(CellUtil.cloneQualifier(cell))+"\t"
+new String(CellUtil.cloneValue(cell),"UTF-8")+"\t"
+cell.getTimestamp());
i++;
}
}
//输出总的行数
System.out.println(i);
ht.close();
}
hbase(main):009:0> scan 'users',{FILTER=>'FirstKeyOnlyFilter()'}
ROW COLUMN+CELL
xiaoming column=address:city, timestamp=1441997498965, value=hangzhou
xiaoming01 column=address:contry, timestamp=1442000277200, value=\xE4\xB8\xAD\xE5\x9B\xBD
xiaoming02 column=info:age, timestamp=1441998917594, value=24
xiaoming03 column=info:age, timestamp=1441998919607, value=24
zhangyifei column=address:city, timestamp=1441997499108, value=jieyang
5 row(s) in 0.0240 seconds
常见过滤器API
package com.aura.hbase.test;
import java.io.IOException;
import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.filter.FilterList;
import org.apache.hadoop.hbase.filter.MultipleColumnPrefixFilter;
import org.apache.hadoop.hbase.filter.PageFilter;
import org.apache.hadoop.hbase.filter.PrefixFilter;
import org.apache.hadoop.hbase.filter.QualifierFilter;
import org.apache.hadoop.hbase.filter.RowFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueExcludeFilter;
import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
import org.apache.hadoop.hbase.filter.SubstringComparator;
import org.apache.hadoop.hbase.filter.ValueFilter;
import org.apache.hadoop.hbase.filter.BinaryComparator;
import org.apache.hadoop.hbase.filter.ColumnPrefixFilter;
import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp;
import org.apache.hadoop.hbase.filter.FamilyFilter;
import org.apache.hadoop.hbase.util.Bytes;
import org.junit.Test;
import com.aura.hbase.utils.HBasePrintUtil;
public class HBaseFilterTest {
public static final String ZOOKEEPER_LIST = "node01:2181,node02:2181,node03:2181";
public static final String TABLE_NAME = "user_info";
public static Configuration conf = null;
public static Admin admin = null;
public static Table table = null;
static {
conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum", ZOOKEEPER_LIST);
try {
Connection conn = ConnectionFactory.createConnection(conf);
admin = conn.getAdmin();
table = conn.getTable(TableName.valueOf(TABLE_NAME));
} catch (IOException e) {
e.printStackTrace();
}
}
/*
* 扫描全表,查寻指定列族的记录
*/
@Test
public void testScanWithFamily() throws Exception {
Scan scan = new Scan();
scan.addFamily("base_info".getBytes());
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 扫描全表,查寻指定列族、指定列的记录
*/
@Test
public void testScanWithColumn() throws Exception {
Scan scan = new Scan();
scan.addColumn("base_info".getBytes(), "name".getBytes());
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 扫描全表,查寻指定时间戳或指定时间戳范围的记录
*/
@Test
public void testScanWithTimestamp() throws Exception {
Scan scan = new Scan();
// 指定时间戳,查出一条
// scan.setTimeStamp(1514443301587L);
// 指定时间戳范围,查出一条或多条
scan.setTimeRange(1514443301340L, 1514443301587L);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 扫描全表,查寻指定rowkey或rowkey范围的记录
*/
@Test
public void testScanWithRowkey() throws Exception {
Scan scan = new Scan();
scan.setStartRow("baiyc_20150716_0003".getBytes());
scan.setStopRow("baiyc_20150716_0005".getBytes());
/*
* 如果只设置了 startRow,就查询从startRow到表末尾的记录(不包括表最后的rowkey所在的那一行记录)
* 如果只设置了 stopRow,就查询从表开头到stopRow的记录(不包括stopRow的那一行记录)
*/
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试RowFilter
* 扫描全表,查寻rowkey小于等于"baiyc_20150716_0003"的记录
*/
@Test
public void testRowFilter() throws Exception {
Scan scan = new Scan();
Filter filter = new RowFilter(CompareOp.LESS_OR_EQUAL, new BinaryComparator(Bytes.toBytes("baiyc_20150716_0003")));
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试FamilyFilter
* 扫描全表,查寻列簇大于"base_info"的记录
*/
@Test
public void testFamilyFilter() throws Exception {
Scan scan = new Scan();
Filter filter = new FamilyFilter(CompareOp.GREATER, new BinaryComparator(Bytes.toBytes("base_info")));
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试QualifierFilter
* 扫描全表,查寻列名等于"name"的记录
*/
@Test
public void testQualifierFilter() throws Exception {
Scan scan = new Scan();
Filter filter = new QualifierFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes("name")));
/*
* BinaryComparator比较器:匹配完全等值的列名
* Filter filter = new QualifierFilter(CompareOp.EQUAL, new BinaryComparator(Bytes.toBytes("name")));
*
* BinaryPrefixComparator比较器:匹配列名的前缀为"na"的记录
* Filter filter = new QualifierFilter(CompareOp.EQUAL, new BinaryPrefixComparator(Bytes.toBytes("na")));
*
* RegexStringComparator比较器:匹配列名满足正则表达式"na."的记录
* Filter filter = new QualifierFilter(CompareOp.EQUAL, new RegexStringComparator("na."));
*/
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试ValueFilter
* 扫描全表,查寻列的值中包含"mus"子串的记录
*/
@Test
public void testValueFilter() throws Exception {
Scan scan = new Scan();
Filter filter = new ValueFilter(CompareOp.EQUAL, new SubstringComparator("mus"));
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试FilterList
* 同时添加多个过滤器
*/
@Test
public void testFilterList() throws Exception {
Scan scan = new Scan();
Filter filter1 = new FamilyFilter(CompareOp.GREATER, new BinaryComparator(Bytes.toBytes("base_info")));
Filter filter2 = new ValueFilter(CompareOp.NOT_EQUAL, new BinaryComparator(Bytes.toBytes("music")));
FilterList list = new FilterList(filter1, filter2);
scan.setFilter(list);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试PageFilter
* 分页过滤器,从指定rowkey开始,显示指定的条数
*/
@Test
public void testPageFilter() throws Exception {
Scan scan = new Scan();
// 设置每页显示4页
Filter filter = new PageFilter(4);
// 设置起始的rowkey
scan.setStartRow("baiyc_20150716_0003".getBytes());
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试SingleColumnValueFilter:单列值过滤器,会返回满足条件的整行
* 扫描全表,查询列族为"base_info",列名为"name",且列值包括"zhangsan"子串的所有行
*/
@Test
public void testSingleColumnValueFilter() throws Exception {
Scan scan = new Scan();
SingleColumnValueFilter filter = new SingleColumnValueFilter(
Bytes.toBytes("base_info"),
Bytes.toBytes("name"),
CompareOp.EQUAL,
new SubstringComparator("zhangsan"));
/*
* 如果不设置为 true,则那些不包含指定column的行也会返回
* 比如,现在有一行它没有"name"这个列,它的所有的列值中也不包括"shangsan"这个子串,那么这一行也会返回
* 设置为true,只会返回那些有"name"这个列,并且满足过滤条件的行
*/
filter.setFilterIfMissing(true);
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试SingleColumnValueExcludeFilter:单列值排除器,返回排除了该列的结果
* 与上面的过滤器查询出来的行相同,但不打印"name"那一列
*/
@Test
public void testSingleColumnValueExcludeFilter() throws Exception {
Scan scan = new Scan();
SingleColumnValueExcludeFilter filter = new SingleColumnValueExcludeFilter(
Bytes.toBytes("base_info"),
Bytes.toBytes("name"),
CompareOp.EQUAL,
new SubstringComparator("zhangsan"));
filter.setFilterIfMissing(true);
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试PrefixFilter:前缀过滤器,针对行键
* 扫描全表:查询rowkey的前缀为"baiyc"的全部行
*/
@Test
public void testPrefixFilter() throws Exception {
Scan scan = new Scan();
Filter filter = new PrefixFilter(Bytes.toBytes("baiyc"));
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试ColumnPrefixFilter:列前缀过滤器
* 扫描全表:查询列名的前缀为"na"的全部记录
*/
@Test
public void testColumnPrefixFilter() throws Exception {
Scan scan = new Scan();
Filter filter = new ColumnPrefixFilter(Bytes.toBytes("na"));
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
/*
* 测试MultipleColumnPrefixFilter:基于列名设置多个前缀过滤数据
* 扫描全表:查询列名的前缀为"na"和列名的前缀为"ag"的全部记录
*/
@Test
public void testMultipleColumnPrefixFilter() throws Exception {
Scan scan = new Scan();
byte[][] prefixes = new byte[][] {Bytes.toBytes("na"), Bytes.toBytes("ag")};
Filter filter = new MultipleColumnPrefixFilter(prefixes);
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
HBasePrintUtil.printResultScanner(scanner);
}
}