xueqiu/rdr

rdr's Issues

请问现在支持redis 4.0以上吗

redis5.0.5的dump文件 报出异常decode rdbfile err: rdb: invalid file format

decode rdbfile err: rdb: invalid RDB version number 10

decode rdbfile err: rdb: invalid RDB version number 10

redis version: 7.0.4

Support for RDB version 7

When I try to parse an RDB file from Redis 3.2.5 I get this error:

start parsing...
decode rdbfile err: rdb: invalid RDB version number 7

Looking at the vendored version of cupcake/rdb it does indeed seem like it's unsupported: https://github.com/xueqiu/rdr/blob/master/vendor/github.com/cupcake/rdb/decoder.go#L588

But the version 7 support landed in upstream 6 months ago: tent/rdb#6

So it does seem like you need to re-vendor and distribute a new build.

decode rdbfile err: rdb: invalid RDB version number 10

是不是该更新了。

fork后, 重新打包了3个平台的包, 原来的下载链接失效了

https://github.com/gohouse/rdr/releases/tag/v0.1.0

Show命令支持读取dump后的json文件

能否提供一个可选参数,根据某个rdb的结果,大致计算出一个集群的key分布情况

如:在命令行提供一个节点数,然后计算出来的结果乘以节点数,就可以得出redis cluster一个集群的数据.该方案有一个缺点不够准确,但是也能满足绝大部分需求.

分析结果不准确，差异较大

redis_version:3.2.11
RDB 文件：1.5G dump.rdb
db0:keys=42300943,expires=42294595,avg_ttl=40692106

分析结果：
"TotleBytes": 5965130248,
"TotleNum": 26611622,
"TypeBytes": {
"hash": 2916,
"set": 35720,
"sortedset": 2383546340,
"string": 3581545272
},
"TypeNum": {
"hash": 1,
"set": 4,
"sortedset": 1,
"string": 26611616
}

Files for readme

能不能添加上key的过期时间？

图形做的非常酷炫！
能不能把key的过期时间添加上？有助于分析哪些key存放太久，甚至设置了永久不过期

out of memory

start parsing...
runtime: out of memory: cannot allocate 227278848-byte block (1839726592 in use)

fatal error: out of memory

runtime stack:
runtime.throw(0x79a639, 0xd)
c:/go/src/runtime/panic.go:616 +0x6b
runtime.largeAlloc(0xd8c0000, 0x12b10001, 0x4)
c:/go/src/runtime/malloc.go:828 +0xdc
runtime.mallocgc.func1()
c:/go/src/runtime/malloc.go:721 +0x39
runtime.systemstack(0x0)
c:/go/src/runtime/asm_386.s:464 +0x5e
runtime.mstart()
c:/go/src/runtime/proc.go:1175

goroutine 1 [running]:
runtime.systemstack_switch()
c:/go/src/runtime/asm_386.s:418 fp=0x12d41ae4 sp=0x12d41ae0 pc=0x44ac30
runtime.mallocgc(0xd8c0000, 0x75aec0, 0x3000101, 0x0)
c:/go/src/runtime/malloc.go:720 +0x67b fp=0x12d41b38 sp=0x12d41ae4 pc=0x
40dd3b
runtime.newarray(0x75aec0, 0x110000, 0x8c5b0332)
c:/go/src/runtime/malloc.go:855 +0x52 fp=0x12d41b4c sp=0x12d41b38 pc=0x4
0dff2
runtime.makeBucketArray(0x741680, 0x401414, 0x35476b40, 0x8c5b0332)
c:/go/src/runtime/hashmap.go:881 +0xbf fp=0x12d41b6c sp=0x12d41b4c pc=0x
40739f
runtime.hashGrow(0x741680, 0x12bc5cc0)
c:/go/src/runtime/hashmap.go:905 +0x6a fp=0x12d41b98 sp=0x12d41b6c pc=0x
4074fa
runtime.mapassign(0x741680, 0x12bc5cc0, 0x12d41c48, 0xbf8e00)
c:/go/src/runtime/hashmap.go:579 +0x262 fp=0x12d41bf8 sp=0x12d41b98 pc=0
x4065b2
main.(*Counter).countByKeyPrefix(0x12b48190, 0x2950a4b0)
E:/redis-rdb-tools/rdr/counter.go:187 +0x11f fp=0x12d41c6c sp=0x12d41bf8
pc=0x6edd2f
main.(*Counter).count(0x12b48190, 0x2950a4b0)
E:/redis-rdb-tools/rdr/counter.go:124 +0x17f fp=0x12d41ca4 sp=0x12d41c6c
pc=0x6ed31f
main.(*Counter).Count(0x12b48190, 0x12b3a440)
E:/redis-rdb-tools/rdr/counter.go:69 +0x3b fp=0x12d41cb8 sp=0x12d41ca4 p
c=0x6ecb3b
main.show(0x12b4e370)
E:/redis-rdb-tools/rdr/main.go:68 +0x281 fp=0x12d41d88 sp=0x12d41cb8 pc=
0x6f0c31
github.com/urfave/cli.HandleAction(0x72b3c0, 0x7b6bf0, 0x12b4e370, 0x0, 0x12b661
80)
C:/GOPATH/src/github.com/urfave/cli/app.go:503 +0x62 fp=0x12d41d98 sp=0x
12d41d88 pc=0x63ca72
github.com/urfave/cli.Command.Run(0x794903, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7add
81, 0x32, 0x0, ...)
C:/GOPATH/src/github.com/urfave/cli/command.go:165 +0x3d7 fp=0x12d41e74
sp=0x12d41d98 pc=0x63d517
github.com/urfave/cli.(*App).Run(0x12b1ca80, 0x12b3a040, 0x5, 0x8, 0x0, 0x0)
C:/GOPATH/src/github.com/urfave/cli/app.go:259 +0x5cc fp=0x12d41f70 sp=0
x12d41e74 pc=0x63b19c
main.main()
E:/redis-rdb-tools/rdr/main.go:146 +0x24d fp=0x12d41fc8 sp=0x12d41f70 pc
=0x6f164d
runtime.main()
c:/go/src/runtime/proc.go:198 +0x1ea fp=0x12d41ff0 sp=0x12d41fc8 pc=0x42
8a1a
runtime.goexit()
c:/go/src/runtime/asm_386.s:1665 +0x1 fp=0x12d41ff4 sp=0x12d41ff0 pc=0x4
4c6f1

goroutine 5 [chan send]:
main.(*Decoder).Set(0x12b9d640, 0x35415270, 0x4b, 0x4b, 0x2d0c9200, 0x208, 0x208
, 0x0, 0x0)
E:/redis-rdb-tools/rdr/decoder.go:61 +0x22f
github.com/cupcake/rdb.(*decode).readObject(0x12b25f58, 0x35415270, 0x4b, 0x4b,
0x0, 0x0, 0x0, 0x0, 0x0)
C:/GOPATH/src/github.com/cupcake/rdb/decoder.go:233 +0xc0
github.com/cupcake/rdb.(*decode).decode(0x12b25f58, 0x1000, 0x1000)
C:/GOPATH/src/github.com/cupcake/rdb/decoder.go:215 +0xe0
github.com/cupcake/rdb.Decode(0x7ea4b0, 0x12b053f8, 0x7ee2c0, 0x12b9d640, 0x0, 0
x0)
C:/GOPATH/src/github.com/cupcake/rdb/decoder.go:67 +0xbf
main.decode(0x12b4e370, 0x12b9d640, 0x12b08100, 0xb)
E:/redis-rdb-tools/rdr/main.go:43 +0xdc
created by main.show
E:/redis-rdb-tools/rdr/main.go:66 +0x266

E:\redis\Redis-x64-3.0.504\WEB_TMP>

环境：win10
rdb大小：4G
机器内存：8G
跑几次，直接爆掉
你说“5GB rdbfile takes about 2mins on my PC”是在个人PC Windows下面吗？ @mengxy

go build 的时候报错

$ go build main.go

github.com/xueqiu/rdr/decoder

decoder\decoder.go:46:16: undefined: rdb.Info

多个rdb文件作为参数时，解析的结果好像不对

页面展示的也会有多个rdb列表，但实际显示好像根据最后一个rdb文件的内存

建议增加显示在几号数据库

decoder.Entry中NumOfElem字段是什么意思？

string类型的NumOfElem为什么不是1？

go test failed long long ago

git reset --hard 7666352

$ go test ./...
zsh: correct './...' to './..' [nyae]? n
--- FAIL: TestGetLargestKeyPrefixes (0.00s)
        Error Trace:    counter_test.go:36
        Error:      	Not equal:
                    	expected: "RELATIONSFOLLOWERIDS"
                    	received: "RELATIONSFOLLOWERIDS0000000000"
FAIL
FAIL	github.com/xueqiu/rdr	1.283s
?   	github.com/xueqiu/rdr/static	[no test files]
?   	github.com/xueqiu/rdr/views	[no test files]
FAIL

If i extract it as a library seperate from main.go, would u accept the PR?

Or I will just fork it for myself

使用教程能否详细一些，谢谢！

你好，刚学习redis，看这个教程有点难，有详细的使用说明吗？

win10家庭版运行rdr-windows.exe后无任何反应

以管理员权限运行rdr-windows.exe文件无任何反应，杀毒软件也已关闭

系统版本：
版本 Windows 10 家庭中文版
版本号 21H2
安装日期 ‎2021/‎8/‎13
操作系统内部版本 19044.1586
体验 Windows Feature Experience Pack 120.2212.4170.0

能否支持 Apple M1 芯片

如题，谢谢

能否添加把rdb统计信息以json的形式dump到STDOUT的功能

这样便于统计信息的存储，增量分析。

是否有windows 版本的release

是否有windows 版本的release

分析8G rdb快照，系统内存64G都不够用

redis 4.0 版本，执行好久也没解析出来，系统内存直到耗尽

怎么更改报表中统计使用的前缀符号啊？

如题，我的不知道我的报表中使用的前缀符号是啥。怎么更改啊 我们系统中用的是 "_"

rdr 下载链接失效了

rdr 可执行文件的下载链接失效了。

decode rdbfile err: rdb: invalid RDB version number 8

decode rdbfile err: rdb: invalid RDB version number 8

redis 4.0版本

performance inquire

I read from this website http://www.infoq.com/cn/articles/analysis-redis that your performance is times better than redis-rdb-tools. I'm wondering , do you have any statistics compare these two. How is performance with 10 millions keys or more. And how can I use it but print results in files instead show on a website

@dongmx hope hear from you soon.

请问 linux arm 如何编译

请问 linux arm如何编译此项目

请问这个项目是如何运行起来的呢

你好，你的项目是用go语言开发的，而我对go语言不太了解，请问这个项目是如何运行起来的呢

Could u add me as collaborator?

样式有问题，阅读性不好，你用的什么浏览器

试了下，top 100 largest keys显示不友好。

rdr show只支持本地监听地址么？

./rdr show -p 8080 *.rdb 只支持本地监听地址么？我在测试机上分析，怎么用自己的机器访问过去呢。

分析9GB的rdb文件时报fatal error: out of memory，本地机器内存32GB

如下为异常详情：

runtime: VirtualAlloc of 4194304 bytes failed with errno=1455
fatal error: out of memory

runtime stack:
runtime.throw(0x7c8633, 0xd)
        /usr/local/go/src/runtime/panic.go:617 +0x64
runtime.sysMap(0x6d800000, 0x400000, 0xd55d18)
        /usr/local/go/src/runtime/mem_windows.go:122 +0x116
runtime.(*mheap).sysAlloc(0xd45940, 0x2000, 0x2, 0x420a29)
        /usr/local/go/src/runtime/malloc.go:633 +0x162
runtime.(*mheap).grow(0xd45940, 0x1, 0x0)
        /usr/local/go/src/runtime/mheap.go:1232 +0x34
runtime.(*mheap).allocSpanLocked(0xd45940, 0x1, 0xd55d28, 0x0)
        /usr/local/go/src/runtime/mheap.go:1150 +0x364
runtime.(*mheap).alloc_m(0xd45940, 0x1, 0x11cb0013, 0x1)
        /usr/local/go/src/runtime/mheap.go:977 +0xd5
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:1048 +0x3e
runtime.systemstack(0x0)
        /usr/local/go/src/runtime/asm_386.s:396 +0x53
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1153

goroutine 19 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_386.s:357 fp=0x11c2acd8 sp=0x11c2acd4 pc=0x44c150
runtime.(*mheap).alloc(0xd45940, 0x1, 0x10013, 0x1)
        /usr/local/go/src/runtime/mheap.go:1047 +0x6c fp=0x11c2acfc sp=0x11c2acd8 pc=0x42080c
runtime.(*mcentral).grow(0xd460f8, 0x0)
        /usr/local/go/src/runtime/mcentral.go:256 +0x74 fp=0x11c2ad20 sp=0x11c2acfc pc=0x4134c4
runtime.(*mcentral).cacheSpan(0xd460f8, 0x38)
        /usr/local/go/src/runtime/mcentral.go:106 +0x340 fp=0x11c2ad5c sp=0x11c2ad20 pc=0x412ff0
runtime.(*mcache).refill(0x11906bc, 0x13)
        /usr/local/go/src/runtime/mcache.go:135 +0x91 fp=0x11c2ad6c sp=0x11c2ad5c pc=0x412a71
runtime.(*mcache).nextFree(0x11906bc, 0x13, 0x11c2adb4, 0x0, 0x7e)
        /usr/local/go/src/runtime/malloc.go:786 +0x7a fp=0x11c2ad88 sp=0x11c2ad6c pc=0x40918a
runtime.mallocgc(0x80, 0x0, 0x0, 0x71d11f)
        /usr/local/go/src/runtime/malloc.go:939 +0x5c3 fp=0x11c2addc sp=0x11c2ad88 pc=0x4098f3
runtime.slicebytetostring(0x0, 0x6d6b1f00, 0x7e, 0x7e, 0x0, 0x0)
        /usr/local/go/src/runtime/string.go:102 +0x86 fp=0x11c2adf0 sp=0x11c2addc pc=0x43cc26
main.(*MemProfiler).ElemLen(0x11c50544, 0x6d6b1f00, 0x7e, 0x7e, 0xa0, 0x0)
        /Users/dongmingxin/go/src/github.com/xueqiu/rdr/memprofiler.go:162 +0x3d fp=0x11c2ae14 sp=0x11c2adf0 pc=0x71d17d
main.(*Decoder).Set(0x11c50540, 0x6d7ed920, 0x2b, 0x2b, 0x6d6b1f00, 0x7e, 0x7e, 0x0, 0x0)
        /Users/dongmingxin/go/src/github.com/xueqiu/rdr/decoder.go:59 +0x187 fp=0x11c2ae68 sp=0x11c2ae14 pc=0x7194c7
github.com/xueqiu/rdr/vendor/github.com/dongmx/rdb.(*decode).readObject(0x11c2af60, 0x6d7ed920, 0x2b, 0x2b, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/dongmingxin/go/src/github.com/xueqiu/rdr/vendor/github.com/dongmx/rdb/decoder.go:322 +0x41d fp=0x11c2aef4 sp=0x11c2ae68 pc=0x68300d
github.com/xueqiu/rdr/vendor/github.com/dongmx/rdb.(*decode).decode(0x11c2af60, 0x1000, 0x1000)
        /Users/dongmingxin/go/src/github.com/xueqiu/rdr/vendor/github.com/dongmx/rdb/decoder.go:302 +0xf2 fp=0x11c2af44 sp=0x11c2aef4 pc=0x682772
github.com/xueqiu/rdr/vendor/github.com/dongmx/rdb.Decode(0x854f90, 0x11c47410, 0x85c540, 0x11c50540, 0x11c47410, 0x0)
        /Users/dongmingxin/go/src/github.com/xueqiu/rdr/vendor/github.com/dongmx/rdb/decoder.go:75 +0xaf fp=0x11c2afac sp=0x11c2af44 pc=0x68258f
main.decode(0x11ca8140, 0x11c50540, 0x11c58090, 0xf)
        /Users/dongmingxin/go/src/github.com/xueqiu/rdr/main.go:44 +0xe9 fp=0x11c2afe0 sp=0x11c2afac pc=0x71b0d9
runtime.goexit()
        /usr/local/go/src/runtime/asm_386.s:1321 +0x1 fp=0x11c2afe4 sp=0x11c2afe0 pc=0x44d971
created by main.dump
        /Users/dongmingxin/go/src/github.com/xueqiu/rdr/main.go:103 +0x184

goroutine 1 [running]:
        goroutine running on other thread; stack unavailable

Why does nobody support this great project

Why does nobody support this great project now? I very like it ,but it is invalid now.Three release link is unworkable.

【功能建议】过期时间key和未设置过期时间key的数量及大小

大佬，请问是否考虑新增功能：分析设置过期时间key和未设置过期时间key的数量及内存占用大小

redis5.0 类型不兼容

使用redis5.0 的阿里云rdb文件报以下错误
err: Not supported load module 8869529002411227136

QUICKLIST

3.2+新引入的encoding为quicklist作为list的基础类型，而quicklist在rdr解析以后NumOfElement很大：18446744073709551615