[转帖]分析redis 大key

转载

jinanxiaolaohu6 2024-01-16 11:34:50

文章标签 redis json 数据库 文章分类 Html/CSS 前端开发

redis-rdb-tools 是一个 python 的解析 rdb 文件的工具，在分析内存的时候，我们主要用它生成内存快照。

主要有以下三个功能：

生成内存快照
转储成 json 格式
使用标准的 diff 工具比较两个 dump 文件

安装

pip install rdbtools python-lzf
# 在安装python-lzf之前，要安装python-devel依赖包，否则安装会失败
 yum install python-devel

rdbtools工具包括了3个可执行文件：

rdb – 解析整个rdb文件
redis-memory-for-key – 解析server里的单个key
redis-profiler –解析rdb文件成html格式

命令help

1、rdb –help：解析整个rdb文件

usage: rdb [options] /path/to/dump.rdb

Example : rdb --command json -k "user.*" /var/redis/6379/dump.rdb

positional arguments:
-- 要处理的dump文件
  dump_file RDB Dump file to process

optional arguments:
-- 帮助
  -h, --help show this help message and exit
-- 要处理的命令，-c后的有效参数为：json, diff,justkeys, justkeyvals, memory，protocol
  -c CMD, --command CMD
                        Command to execute. Valid commands are json, diff,
                        justkeys, justkeyvals, memory and protocol
-- 输出文件
  -f FILE, --file FILE Output file
-- 数据库号，可以提供多个数据库。如果未指定，则包括所有数据库。
  -n DBS, --db DBS Database Number. Multiple databases can be provided.
                        If not specified, all databases will be included.
-- 要导出的key。这可以是一个正则表达式
  -k KEYS, --key KEYS Keys to export. This can be a regular expression
-- key不导出。这可以是一个正则表达式
  -o NOT_KEYS, --not-key NOT_KEYS
                        Keys Not to export. This can be a regular expression
-- 解析的数据类型。可能的值为string，hash，set，sortedset，list。可以输入多种类型提供。如果未指定，则为所有数据类型
  -t TYPES, --type TYPES
                        Data types to include. Possible values are string,
                        hash, set, sortedset, list. Multiple typees can be
                        provided. If not specified, all data types will be
                        returned
-- 将key的内存输出限制为大于或等此值（以字节为单位）
  -b BYTES, --bytes BYTES
                        Limit memory output to keys greater to or equal to
                        this value (in bytes)
-- 将内存按大小输出前N个key
  -l LARGEST, --largest LARGEST
                        Limit memory output to only the top N keys (by size)
-- 将字符串转义为编码：raw（默认），print，utf8或base64。
  -e {raw,print,utf8,base64}, --escape {raw,print,utf8,base64}
                        Escape strings to encoding: raw (default), print,
                        utf8, or base64.
-- 使用command protocol参数，从所有键中删除到期的key
  -x, --no-expire With protocol command, remove expiry from all keys
-- 使用command protocol参数，将N秒添加到key的到期时间
  -a N, --amend-expire N
                        With protocol command, add N seconds to key expiry
                        time

2，redis-memory-for-key –help：– 解析server里指定的单个key

Usage: redis-memory-for-key [options] redis-key
Examples :
redis-memory-for-key user:13423
redis-memory-for-key -s localhost -p 6379 user:13423


Options:
-- 帮助
  -h, --help            show this help message and exit
-- 服务地址，默认127.0.0.1
  -s HOST, --server=HOST
                        Redis Server hostname. Defaults to 127.0.0.1
-- 服务端口，默认6379                        
  -p PORT, --port=PORT Redis Server port. Defaults to 6379
--服务密码
  -a PASSWORD, --password=PASSWORD
                        Password to use when connecting to the server
-- 数据库号，默认0
  -d DB, --db=DB Database number, defaults to 0

3、redis-profiler –help：

Usage: redis-profiler [options] /path/to/dump.rdb

Example 1 : redis-profiler -k "user.*" -k "friends.*" -f memoryreport.html /var/redis/6379/dump.rdb
Example 2 : redis-profiler /var/redis/6379/dump.rdb

Options:
-- 帮助
  -h, --help            show this help message and exit
-- 输出
  -f FILE, --file=FILE Output file
-- 组合在一起的键。多个正则表达式
  -k KEYS, --key=KEYS Keys that should be grouped together. Multiple regexes
                        can be provided

使用

每次运行以上工具时都需要指定一个命令，以指示对解析的RDB数据应执行的操作。操作有：

转储的JSON

rdb -c json test-redis.rdb -f test-redis.json

[root@dev-tools ~]# rdb -c json test-redis.rdb  -f  test-redis.json
[root@dev-tools ~]# head -10 test-redis.json 
[{
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953182b16ad6d-8ca5-4f0d-8e03-94274a43c666,3a34cb89-dbed-4de4-a5f1-620c09f35053":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060514250009392992486,2022060514250009403000370":"172800",
"yundt-cube-center-trade-test:_dtmqconsume/ORDERID_a20cae4d-bf89-44d8-9552-80b27ecd9241":"1",
"yundt-cube-center-scheduler-test:_dtmqconsume/ORDERID_f32d75ca-783b-4e19-8019-4975f82fc0ea":"1",
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060323150007293702486,2022060323150007303710370":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT129691703147679531890b60ff9-1b25-44e5-90ea-337fb28a31c7,2d4e7fd0-37b9-414e-afef-752266bd2507":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022061617500009907132486,2022061617500009907140370":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT129691728607209669772b92b0e-7010-4255-bca2-ae244c32b8cf":"172800",
"yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953183092cef0-d220-416b-b405-298247fbfeee,a11e828b-8849-4f76-8b80-762732b90494":"172800",

过滤解析

正则表达式匹配key，并且仅打印键和值：

rdb -c justkeyvals --key "yundt-cube-center-inventory-test.*" test-redis.rdb

[root@dev-tools ~]# rdb -c  justkeyvals --key "yundt-cube-center-inventory-test.*"  test-redis.rdb  >demo.txt
[root@dev-tools ~]# head -10 demo.txt 

yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953182b16ad6d-8ca5-4f0d-8e03-94274a43c666,3a34cb89-dbed-4de4-a5f1-620c09f35053 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060514250009392992486,2022060514250009403000370 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060323150007293702486,2022060323150007303710370 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT129691703147679531890b60ff9-1b25-44e5-90ea-337fb28a31c7,2d4e7fd0-37b9-414e-afef-752266bd2507 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022061617500009907132486,2022061617500009907140370 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT129691728607209669772b92b0e-7010-4255-bca2-ae244c32b8cf 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953183092cef0-d220-416b-b405-298247fbfeee,a11e828b-8849-4f76-8b80-762732b90494 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT1296917031476795318516d5a98-330a-44a2-abcb-f052398aa0da,2ee21658-46f3-417d-a5cc-5997e3dbcc7f 172800,
yundt-cube-center-inventory-test:INVENTORY_COUNT1296917031476795318ad9ec3a8-06db-4e1b-b566-61eb62779c90,b6837582-ce51-4509-9e8b-34a1c8aef520 172800,

生成内存报告

使用-c memory 运行会生成CSV报告，其中包含该键使用的近似内存。–bytes C 和 –largest N 可用于将输出限制为大于C字节的键或N个最大键。

rdb -c memory test-redis.rdb --bytes 128 -f redis-mem.csv

[root@dev-tools ~]# rdb -c memory test-redis.rdb --bytes 128 -f redis-mem.csv 
[root@dev-tools ~]# head -10 redis-mem.csv 
database,type,key,size_in_bytes,encoding,num_elements,len_largest_element,expiry
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953182b16ad6d-8ca5-4f0d-8e03-94274a43c666,3a34cb89-dbed-4de4-a5f1-620c09f35053,200,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060514250009392992486,2022060514250009403000370,168,string,8,8,
0,string,yundt-cube-center-trade-test:_dtmqconsume/ORDERID_a20cae4d-bf89-44d8-9552-80b27ecd9241,168,string,8,8,2022-10-09T20:39:00.057000
0,string,yundt-cube-center-scheduler-test:_dtmqconsume/ORDERID_f32d75ca-783b-4e19-8019-4975f82fc0ea,168,string,8,8,2022-10-09T21:34:00.100000
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022060323150007293702486,2022060323150007303710370,168,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT129691703147679531890b60ff9-1b25-44e5-90ea-337fb28a31c7,2d4e7fd0-37b9-414e-afef-752266bd2507,200,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969167669986651372022061617500009907132486,2022061617500009907140370,168,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT129691728607209669772b92b0e-7010-4255-bca2-ae244c32b8cf,152,string,8,8,
0,string,yundt-cube-center-inventory-test:INVENTORY_COUNT12969170314767953183092cef0-d220-416b-b405-298247fbfeee,a11e828b-8849-4f76-8b80-762732b90494,200,string,8,8,

生成的CSV具有以下列：

database：数据库编号
type：数据类型
key：键
size_in_bytes：使用的内存：包括键，值和任何其他开销
encoding：RDB编码类型
num_elements：key中的value的个数
len_largest_element：key中的value的长度
expiry：过期时间

注意：内存使用情况是近似的。通常，实际使用的内存将略高于报告的内存。可以按键或数据库编号或数据类型过滤报告。内存报告应有助于检测由应用程序逻辑引起的内存泄漏。它还将帮助优化Redis的内存使用。

对key按内存大小排序

sort -t ',' -k4 -rn redis-mem.csv  >redis-mem-sort.csv

查找单键使用的内存

查找特定键使用的内存（运行整个内存报告非常耗时），使用redis-memory-for-key

redis-memory-for-key person:1
redis-memory-for-key -s localhost -p 6379 -a mypassword person:1
Key person:1
Bytes 111
Type hash
Encoding ziplist
Number of Elements 2
Length of Largest Element 8

作者：Mister Li