Giter Site home page Giter Site logo

redis-port's Introduction

redis-port

Build Status

parse redis rdb file, sync data between redis master and slave

  • DECODE dumped payload to human readable format (hex-encoding)
redis-port decode    [--ncpu=N] [--parallel=M] \
    [--input=INPUT] \
    [--output=OUTPUT]
  • RESTORE rdb file to target redis
redis-port restore   [--ncpu=N] [--parallel=M] \
    [--input=INPUT]  [--faketime=FAKETIME] [--extra] [--filterdb=DB] \
     --target=TARGET [--auth=AUTH] [--redis|--codis]
  • DUMP rdb file from master redis
redis-port dump      [--ncpu=N] [--parallel=M] \
     --from=MASTER   [--password=PASSWORD] [--extra] \
    [--output=OUTPUT]
  • SYNC data from master to slave
redis-port sync      [--ncpu=N] [--parallel=M] \
     --from=MASTER   [--password=PASSWORD] [--psync] [--filterdb=DB] \
     --target=TARGET [--auth=AUTH] [--redis|--codis] [--sockfile=FILE [--filesize=SIZE]]

Options

  • -n N, --ncpu=N

set runtime.GOMAXPROCS to N

  • -p M, --parallel=M

set number of parallel routines

  • -i INPUT, --input=INPUT

use INPUT as input file, or if it is not given, redis-port reads from stdin (means '/dev/stdin')

  • -o OUTPUT, --output=OUTPUT

use OUTPUT as output file, or if it is not given, redis-port writes to stdout (means '/dev/stdout')

  • -m MASTER, --master=MASTER

specify the master redis

  • -t TARGET, --target=TARGET

specify the slave redis (or target redis)

  • -P PASSWORD, --password=PASSWORD

specify the redis auth password

  • -A AUTH, --auth=AUTH

specify the auth password for target

  • -e, --extra

dump or restore following redis backlog commands

  • --redis

target is normal redis instance, default value is false.

  • --codis

target is codis proxy, default value is true.

  • --filterdb=DB

filter specifed db number, default value is '*'

Examples

  • DECODE
$ cat dump.rdb | ./redis-port decode 2>/dev/null
 {"db":0,"type":"string","key":"a","value":"hello"}
 {"db":1,"type":"string","key":"a","value":"9"}
 {"db":0,"type":"hash","key":"c","field":"hello","value":"world"}
 {"db":0,"type":"expire","key":"c","expireat":1487663341422}
 {"db":0,"type":"list","key":"b","index":0,"value":"hello"}
 {"db":0,"type":"list","key":"b","index":1,"value":"world"}
 {"db":0,"type":"zset","key":"d","member":"hello","score":1}
 {"db":0,"type":"zset","key":"d","member":"world","score":1.8}
  ... ...
  • RESTORE
$ ./redis-port restore -i dump.rdb -t 127.0.0.1:6379 -n 8
  2014/10/28 15:08:26 [ncpu=8] restore from 'dump.rdb' to '127.0.0.1:6379'
  2014/10/28 15:08:27 total = 280149161 -     14267777 [  5%]
  2014/10/28 15:08:28 total = 280149161 -     27325530 [  9%]
  2014/10/28 15:08:29 total = 280149161 -     40670677 [ 14%]
  ... ...                                                    
  2014/10/28 15:08:47 total = 280149161 -    278070563 [ 99%]
  2014/10/28 15:08:47 total = 280149161 -    280149161 [100%]
  2014/10/28 15:08:47 done
  • DUMP
$ ./redis-port dump -f 127.0.0.1:6379 -o save.rdb
  2014/10/28 15:12:05 [ncpu=1] dump from '127.0.0.1:6379' to 'save.rdb'
  2014/10/28 15:12:06 -
  2014/10/28 15:12:07 -
  ... ...
  2014/10/28 15:12:10 total = 278110192 -            0 [  0%]
  2014/10/28 15:12:11 total = 278110192 -    278110192 [100%]
  2014/10/28 15:12:11 done

$ ./redis-port dump -f 127.0.0.1:6379 | tee save.rdb | ./redis-port decode -o save.log -n 8 2>/dev/null
  2014/10/28 15:12:55 [ncpu=1] dump from '127.0.0.1:6379' to '/dev/stdout'
  2014/10/28 15:12:56 -
  ... ...
  2014/10/28 15:13:10 total = 278110192  -   264373070 [  0%]
  2014/10/28 15:13:11 total = 278110192  -   278110192 [100%]
  2014/10/28 15:13:11 done
  • SYNC
$ ./redis-port sync -f 127.0.0.1:6379 -t 127.0.0.1:6380 -n 8
  2014/10/28 15:15:41 [ncpu=8] sync from '127.0.0.1:6379' to '127.0.0.1:6380'
  2014/10/28 15:15:42 -
  2014/10/28 15:15:43 -
  2014/10/28 15:15:44 -
  2014/10/28 15:15:46 total = 278110192 -      9380927 [  3%]
  2014/10/28 15:15:47 total = 278110192 -     18605075 [  6%]
  ... ...                                              [    ]
  2014/10/28 15:16:14 total = 278110192 -    269990892 [ 97%]
  2014/10/28 15:16:15 total = 278110192 -    278110192 [100%]
  2014/10/28 15:16:15 done
  2014/10/28 15:16:17 pipe: send = 0             recv = 0
  2014/10/28 15:16:18 pipe: send = 0             recv = 0
  ... ...

redis-port's People

Contributors

c4pt0r avatar spinlock avatar zonesan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

redis-port's Issues

codis集群 3.2往Redis单实例 2.8版本迁移出错

redis-port版本:
version: 2017-05-02 12:36:19 +0800 @79e0f6ca38fbaf6f5c36ea704cf07a96ce9e9d79 @v1.2.1-2-g79e0f6c
compile: 2017-08-24 20:25:19 +0800 by go version go1.8.3 linux/amd64

命令:
redis-port sync --ncpu=2 --parallel=12 --from=127.0.0.1:7384 --password=****** --psync --target=127.0.0.1:7397 --auth=****** --redis --sockfile=buffer_7384.tmp --filesize=8gb

结果:
2018/09/19 11:40:03 utils.go:216: [PANIC] RESTORE command error
[error]: ERR wrong number of arguments for 'restore' command
[stack]:
1 /data/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:216
main.restoreRdbEntry
0 /data/gopath/src/github.com/CodisLabs/redis-port/cmd/sync.go:212
main.(*cmdSync).SyncRDBFile.func1.1
... ...

无法恢复数据

/data/package/redis-port/redis-port restore -i save.rdb -t 192.168.30.200:6380
2018/04/18 17:53:20 main.go:189: [INFO] set ncpu = 4, parallel = 4
2018/04/18 17:53:20 restore.go:53: [INFO] restore from 'save.rdb' to '192.168.30.200:6380'
2018/04/18 17:53:20 utils.go:211: [PANIC] SLOTSRESTORE command error
[error]: ERR unknown command 'SLOTSRESTORE'
[stack]:
1 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:211
main.restoreRdbEntry
0 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/restore.go:102
main.(*cmdRestore).RestoreRDBFile.func1.1
... ...

make

/bin/sh: 2: ./autogen.sh: not found
Makefile:59: recipe for target 'jemalloc' failed
make[1]: *** [jemalloc] Error 127
Makefile:63: recipe for target 'build-jemalloc' failed
make: *** [build-jemalloc] Error 2

use of closed network connection

错误信息如下

[error]: read tcp 172.16.1.42:42002: use of closed network connection
[stack]:
    1   /root/disaster-recover/cmd/utils.go:223
            main.iocopy
    0   /root/disaster-recover/cmd/main.go:484
            main.func·007

不是每次都出现这个错误 ,偶尔出现。
484行就是如下的一段代码:

481 go func() {
482     p := make([]byte, ReaderBufferSize)
483     for {
484         iocopy(c, ioutil.Discard, p, len(p))
485     }
486 }()

Redis cluster支持问题

请问redis-port支不支持redis cluster呢?比如:cluster to cluster的迁移支不支持?README没看到这方面的说明呢!

运行redis-port,一直报错panic: runtime error

redis版本2.8.19
go版本1.6.3

./redis-port sync -f 127.0.0.1:6379 -t 127.0.0.1:6381 -n 1
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x8181ffb]

goroutine 1 [running]:
panic(0x8214460, 0x18528030)
/usr/local/go/src/runtime/panic.go:481 +0x326
sync/atomic.LoadUint64(0x1855c104, 0x805c9d2, 0x8249610)
/usr/local/go/src/sync/atomic/asm_386.s:159 +0xb
github.com/CodisLabs/redis-port/pkg/libs/log.(_LogLevel).Test(0x1855c104, 0x4, 0x0, 0x18d414d0)
/home/leechanx/go/src/github.com/CodisLabs/redis-port/pkg/libs/log/log.go:72 +0x21
github.com/CodisLabs/redis-port/pkg/libs/log.(_Logger).isDisabled(0x1855c0f0, 0x4, 0x0, 0x0)
/home/leechanx/go/src/github.com/CodisLabs/redis-port/pkg/libs/log/log.go:167 +0x45
github.com/CodisLabs/redis-port/pkg/libs/log.Infof(0x8258ff0, 0x13, 0x0, 0x0, 0x0)
/home/leechanx/go/src/github.com/CodisLabs/redis-port/pkg/libs/log/log.go:529 +0x3e
main.main()
/home/leechanx/redis-port/cmd/main.go:176 +0x1136

port data eror

I use redis-port port data from redis to codis and some error like below:
2017/03/16 11:02:25 sync.go:235: [INFO] total=2908349195 - 2864928461 [ 98%] entry=5111118
2017/03/16 11:02:26 sync.go:235: [INFO] total=2908349195 - 2868982634 [ 98%] entry=5120844
2017/03/16 11:02:27 sync.go:235: [INFO] total=2908349195 - 2873281677 [ 98%] entry=5130978
2017/03/16 11:02:28 sync.go:235: [INFO] total=2908349195 - 2878549522 [ 98%] entry=5141108
2017/03/16 11:02:29 sync.go:235: [INFO] total=2908349195 - 2884614954 [ 99%] entry=5157670
2017/03/16 11:02:30 sync.go:235: [INFO] total=2908349195 - 2891919471 [ 99%] entry=5172628
2017/03/16 11:02:31 sync.go:235: [INFO] total=2908349195 - 2898386154 [ 99%] entry=5185882
2017/03/16 11:02:32 sync.go:235: [INFO] total=2908349195 - 2903861685 [ 99%] entry=5196923
2017/03/16 11:02:33 sync.go:235: [INFO] total=2908349195 - 2908300354 [ 99%] entry=5207640
2017/03/16 11:02:33 sync.go:235: [INFO] total=2908349195 - 2908349187 [ 99%] entry=5211041
2017/03/16 11:02:33 sync.go:237: [INFO] sync rdb done
2017/03/16 11:02:33 sync.go:259: [PANIC] parse command arguments failed
[error]: expect Array, but got <*redis.BulkBytes>
2 /usr/local/share/go/src/github.com/CodisLabs/redis-port/pkg/redis/resp.go:163
github.com/CodisLabs/redis-port/pkg/redis.AsArray
1 /usr/local/share/go/src/github.com/CodisLabs/redis-port/pkg/redis/handler.go:88
github.com/CodisLabs/redis-port/pkg/redis.ParseArgs
0 /usr/local/share/go/src/github.com/CodisLabs/redis-port/cmd/sync.go:258
main.(*cmdSync).SyncCommand.func2
this redis version is : redis_version:3.0.6
I has read this issue:#33
and hope some one can give me some help ,thanks

这是什么问题

2017/09/13 22:05:02 utils.go:365: [PANIC] parse rdb entry error
[error]: EOF
10 /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:74
github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).Read
9 /home/travis/.gimme/versions/go1.7.5.linux.amd64/src/io/io.go:523
io.(*teeReader).Read
8 /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:72
github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).Read
7 /home/travis/.gimme/versions/go1.7.5.linux.amd64/src/io/io.go:307
io.ReadAtLeast
6 /home/travis/.gimme/versions/go1.7.5.linux.amd64/src/io/io.go:325
io.ReadFull
5 /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:243
github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readFull
4 /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:249
github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readBytes
3 /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:179
github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readString
2 /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:132
github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readObjectValue
1 /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/loader.go:146
github.com/CodisLabs/redis-port/pkg/rdb.(*Loader).NextBinEntry
0 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:364
main.newRDBLoader.func1
... ...
[stack]:
0 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:365
main.newRDBLoader.func1

迁移的时候异常

大神们好! 首先感谢这个工具让我有机会能把数据轻松迁移到codis。

最近做迁移的时候遇到一个异常 详细日志如下

2017/10/20 17:06:12 sync.go:235: [INFO] total=3797108680 -    550792228 [ 14%]  entry=5794370
2017/10/20 17:06:13 sync.go:235: [INFO] total=3797108680 -    551975102 [ 14%]  entry=5810504
2017/10/20 17:06:14 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:15 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:16 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:17 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:18 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:19 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:20 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:21 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:22 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:23 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:24 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:25 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:26 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:27 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:28 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:29 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:30 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:31 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:32 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:33 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:34 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:35 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:36 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:37 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:38 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:39 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:40 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:41 sync.go:235: [INFO] total=3797108680 -    552631996 [ 14%]  entry=5819927
2017/10/20 17:06:42 utils.go:365: [PANIC] parse rdb entry error
[error]: EOF
    7   /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:74
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).Read
    6   /home/travis/.gimme/versions/go1.7.5.linux.amd64/src/io/io.go:307
            io.ReadAtLeast
    5   /home/travis/.gimme/versions/go1.7.5.linux.amd64/src/io/io.go:325
            io.ReadFull
    4   /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:243
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readFull
    3   /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:249
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readBytes
    2   /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:157
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readString
    1   /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/loader.go:142
            github.com/CodisLabs/redis-port/pkg/rdb.(*Loader).NextBinEntry
    0   /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:364
            main.newRDBLoader.func1
        ... ...
[stack]:
    0   /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:365
            main.newRDBLoader.func1

重试过好几次了,依旧这样

WRONGTYPE Operation against a key holding the wrong kind of value

2018/04/27 10:59:50 sync.go:103: [INFO] +
2018/04/27 10:59:51 sync.go:109: [INFO] sync: runid = "969c3a1e19ca54b66b2ad780d60edd43472a3a09", offset = 519121821
2018/04/27 10:59:51 sync.go:111: [INFO] sync: rdb file = 1663327129 (1.55gb)
2018/04/27 10:59:51 sync.go:208: [INFO] sync: (r/f,s/f,s) = (read,rdb.forward,rdb.skip/rdb.forward,rdb.skip)
2018/04/27 10:59:52 sync.go:250: [INFO] sync: rdb = 1663327129 - [ 24.74%] (r/f,s/f,s)=(786432/7628,0/0,0) ~ (768.00kb/-,-/-,-) ~ speed=(768.00kb/7628,0/0,0)
2018/04/27 10:59:53 sync.go:250: [INFO] sync: rdb = 1663327129 - [ 48.86%] (r/f,s/f,s)=(1441792/14493,0/0,0) ~ (1.38mb/-,-/-,-) ~ speed=(640.00kb/6865,0/0,0)
2018/04/27 10:59:54 sync.go:250: [INFO] sync: rdb = 1663327129 - [ 67.32%] (r/f,s/f,s)=(2752512/18423,0/0,0) ~ (2.62mb/-,-/-,-) ~ speed=(1.25mb/3930,0/0,0)
2018/04/27 10:59:54 libs.go:462: [PANIC] fetch redigo reply failed
[error]: WRONGTYPE Operation against a key holding the wrong kind of value
[stack]:
2 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/libs.go:462
main.redigoGetResponse
1 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/libs.go:588
main.doRestoreDBEntry.func3
0 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/libs.go:314
main.(*Job).Run.func1
... ...

做同步的时候报错WRONGTYPE Operation against a key holding the wrong kind of value

Twemproxy 迁移数据到codis报错

当我使用./redis-port sync --parallel=1 --from=192.168.81.170:12121 --password=XXX --target=192.168.81.220:19000 --auth=XXXX 命令将数据从Twemproxy 迁移到codis报如下错误:
2018/08/01 12:30:17 main.go:189: [INFO] set ncpu = 4, parallel = 4
2018/08/01 12:30:17 sync.go:56: [INFO] sync from '192.168.81.170:12121' to '192.168.81.220:19000'
2018/08/01 12:30:18 utils.go:109: [PANIC] read sync response = ''
[error]: EOF
[stack]:
0 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:109
main.waitRdbDump.func1
... ...
请问是什么原因,怎么解决?用的版本是redis-port-v1.2.1-go1.7.5-linux。

redis-port 命令行解析错误

redis-port 解析命令行,按照正确的usage输入命令,但是正常功能无法执行,总是输出
Usage:
redis-port decode [--ncpu=N] [--parallel=M] [--input=INPUT] [--output=OUTPUT]
redis-port restore [--ncpu=N] [--parallel=M] [--input=INPUT] --target=TARGET [--auth=AUTH] [--extra] [--faketime=FAKETIME] [--filterdb=DB] ...
查阅最新的代码,
源码文件中 redis-port/blob/master/cmd/main.go 85行 d, err := docopt.Parse(usage, nil, true, "", false)
docopt.Parse第二个参数为nil 是不是有问题,觉得应该是 os.Args ???

redis4.0.1迁移到codis3.2.2报错

2018/07/19 14:34:08 sync.go:123: [INFO] +
2018/07/19 14:34:09 sync.go:73: [INFO] rdb file = 4920652048
2018/07/19 14:34:09 utils.go:361: [PANIC] parse rdb header error
[error]: verify version, invalid RDB version number 8
1 /home/travis/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/loader.go:44
github.com/CodisLabs/redis-port/pkg/rdb.(*Loader).Header
0 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:360
main.newRDBLoader.func1
... ...
[stack]:
0 /home/travis/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:361
main.newRDBLoader.func1
... ...
目前不支持4.x迁移?只是单个实例的redis 啊,没用cluster

为什么redis-port出现连不上redis的时候也要退出

我做了一个测试,在执行sync的时候,把redis弄挂掉,结果redis-port退出了。如果codis出现连不上zk退出还有一点点原因的话,redis-port执行数据同步的时候出现redis连接异常也要退出进程就实在无法理解了。为什么不可以在出现异常的时候仅仅打错误日志,每隔一段时间不停地尝试连接redis,不要杀掉进程了呢?

redis使用redis-port迁移至codis时报错

[root@go_cache bin]# ./redis-port sync -f 127.0.0.1:6379 -t 172.31.12.92:6499 -n 8
2015/05/24 08:30:07 [INFO] set ncpu = 8, parallel = 8
2015/05/24 08:30:07 [INFO] sync from '127.0.0.1:6379' to '172.31.12.92:6499'
2015/05/24 08:30:08 [INFO] rdb file = 1653523
2015/05/24 08:30:08 [PANIC] parse rdb checksum error
[error]: checksum validation failed
    1   /data/rpmbuild/rpmbuild/project/redis-port/BUILD/src/github.com/wandoulabs/redis-port/pkg/rdb/loader.go:51
            github.com/wandoulabs/redis-port/pkg/rdb.(*Loader).Footer
    0   /data/rpmbuild/rpmbuild/project/redis-port/BUILD/src/github.com/wandoulabs/redis-port/cmd/utils.go:184
            main.func·024
        ... ...
[stack]:
    0   /data/rpmbuild/rpmbuild/project/redis-port/BUILD/src/github.com/wandoulabs/redis-port/cmd/utils.go:185
            main.func·024
        ... ...
[root@go_cache bin]#

出现以上错误就中断了

同步后,内存和key数量不一致

master的内存使用是9G,440W的key,用这个工具同步到其他redis上,显示使用内存只有6G,400W的key。
少了几个内存使用和40W的key。
也简单的验证了一些key,都是存在的。 但是还是不明白差的地方在哪,是否能保证数据一致?

Error when sync or dump

2016/08/02 14:04:33 [INFO] set ncpu = 4, parallel = 4
2016/08/02 14:04:33 [INFO] sync from '10.1.101.168:6379' to '10.1.100.219:7018'
2016/08/02 14:04:34 [INFO] -
2016/08/02 14:04:35 [INFO] -
2016/08/02 14:04:36 [INFO] -
2016/08/02 14:04:37 [INFO] -
2016/08/02 14:04:38 [INFO] -
2016/08/02 14:04:39 [INFO] -
2016/08/02 14:04:39 [PANIC] invalid sync response = '$EOF:d7a06afab211ab2508092e1c019e4cdbac1908bb
', n = 0
[error]: strconv.ParseInt: parsing "EOF:d7a06afab211ab2508092e1c019e4cdbac1908bb": invalid syntax
[stack]:
0 /usr/lib/golang/src/github.com/CodisLabs/redis-port/cmd/utils.go:122
main.waitRdbDump.func1

多 DB 的 port

目前 redis-port 似乎只可以支持将原 redis db 0 的数据同步到 codis。
是否可以支持将 Redis 其他 db 的数据同步到 codis, 如将 原 db1 的数据同步到 codis db 0

Target key is busy

迁移数据从codis到redis时,报错 panic Target key is busy,请问这是什么原因,怎么解决?谢谢

decode 出现这个,是什么原因

[root@92780d52277f 6379]# ../../cmd decode --n=2 --i=./dump.rdb --o=test
2017/08/21 09:12:56 main.go:189: [INFO] set ncpu = 2, parallel = 2
2017/08/21 09:12:56 decode.go:45: [INFO] decode from './dump.rdb' to 'test'
2017/08/21 09:12:56 utils.go:361: [PANIC] parse rdb header error
[error]: verify version, invalid RDB version number 8
1 /home/gopath/src/github.com/CodisLabs/redis-port/pkg/rdb/loader.go:44
github.com/CodisLabs/redis-port/pkg/rdb.(*Loader).Header
0 /home/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:360
main.newRDBLoader.func1
... ...
[stack]:
0 /home/gopath/src/github.com/CodisLabs/redis-port/cmd/utils.go:361
main.newRDBLoader.func1

redis-port跑不满cpu

32核的机器,用-n 24 -p 1024 但是cpu、网卡、硬盘都没瓶颈,所以导入速度还不是很理想,是这些参数还可以加大么?

迁移过程中失败

redis源版本
redis_version:2.8.4
迁移到codis。
文件差不多有5个G

2015/09/10 12:39:00 [PANIC] parse rdb entry error

7   /usr/local/codis/src/github.com/wandoulabs/redis-port/pkg/rdb/reader.go:75
        github.com/wandoulabs/redis-port/pkg/rdb.(*rdbReader).Read
6   /usr/local/go/src/io/io.go:298
        io.ReadAtLeast
5   /usr/local/go/src/io/io.go:316
        io.ReadFull
4   /usr/local/codis/src/github.com/wandoulabs/redis-port/pkg/rdb/reader.go:234
        github.com/wandoulabs/redis-port/pkg/rdb.(*rdbReader).readFull
3   /usr/local/codis/src/github.com/wandoulabs/redis-port/pkg/rdb/reader.go:240
        github.com/wandoulabs/redis-port/pkg/rdb.(*rdbReader).readBytes
2   /usr/local/codis/src/github.com/wandoulabs/redis-port/pkg/rdb/reader.go:170
        github.com/wandoulabs/redis-port/pkg/rdb.(*rdbReader).readString
1   /usr/local/codis/src/github.com/wandoulabs/redis-port/pkg/rdb/loader.go:125
        github.com/wandoulabs/redis-port/pkg/rdb.(*Loader).NextBinEntry
0   /usr/local/codis/src/github.com/wandoulabs/redis-port/cmd/utils.go:247
        main.newRDBLoader.func1
    ... ...

[stack]:
0 /usr/local/codis/src/github.com/wandoulabs/redis-port/cmd/utils.go:248
main.newRDBLoader.func1

ERR wrong number of arguments for 'restore' command

./redis-port restore --input=/root/dump1.rdb --target=10.159.3.16:6379 --auth=abcabc --redis
2017/09/14 15:13:09 main.go:189: [INFO] set ncpu = 2, parallel = 2
2017/09/14 15:13:09 restore.go:53: [INFO] restore from '/root/dump1.rdb' to '10.159.3.16:6379'
2017/09/14 15:13:09 utils.go:216: [PANIC] RESTORE command error
[error]: ERR wrong number of arguments for 'restore' command

i followed the help information ,but failed . why?
redis-port restore [--ncpu=N] [--parallel=M] [--input=INPUT] [--faketime=FAKETIME] [--extra] [--filterdb=DB] --target=TARGET [--auth=AUTH] [--redis|--codis]

redis-port 导入数据时出错

rdb文件有800m,使用下面参数导入, 前面是的redi ,后面的codis-proxy

帮我看下是什么原因导致的。

2015/07/15 16:32:13 [INFO] total=758189300 - 168525824 [ 22%] entry=481532
2015/07/15 16:32:14 [INFO] total=758189300 - 168525824 [ 22%] entry=481532
2015/07/15 16:32:14 [PANIC] parse rdb entry error
[error]: EOF
7 /home/test/op_tools/src/github.com/wandoulabs/redis-port/pkg/rdb/reader.go:75
github.com/wandoulabs/redis-port/pkg/rdb.(_rdbReader).Read
6 /usr/local/go/src/io/io.go:298
io.ReadAtLeast
5 /usr/local/go/src/io/io.go:316
io.ReadFull
4 /home/test/op_tools/src/github.com/wandoulabs/redis-port/pkg/rdb/reader.go:234
github.com/wandoulabs/redis-port/pkg/rdb.(_rdbReader).readFull
3 /home/test/op_tools/src/github.com/wandoulabs/redis-port/pkg/rdb/reader.go:240
github.com/wandoulabs/redis-port/pkg/rdb.(_rdbReader).readBytes
2 /home/test/op_tools/src/github.com/wandoulabs/redis-port/pkg/rdb/reader.go:148
github.com/wandoulabs/redis-port/pkg/rdb.(_rdbReader).readString
1 /home/test/op_tools/src/github.com/wandoulabs/redis-port/pkg/rdb/loader.go:125
github.com/wandoulabs/redis-port/pkg/rdb.(*Loader).NextBinEntry
0 /home/test/op_tools/src/github.com/wandoulabs/redis-port/cmd/utils.go:247
main.func·026
... ...
[stack]:
0 /home/test/op_tools/src/github.com/wandoulabs/redis-port/cmd/utils.go:248
main.func·026
... ...

rdb version 7

redis 3.2开始使用quickList类型来存储了,而且不再兼容前面的版本,会支持rdb version 7吗?

Can not solve migrating multiple db to single target db

I want to migrate Redis to Codis(good job dude) and I have multiple db on my redis server. But Codis only use db0. So I just want to migrate multiple db to single target db. Maybe should add a command to deal with this demand.

迁移大list问题

大神们好! ,在迁移过程中出现了一个错误 如下:

2017/02/15 18:25:55 [PANIC] parse command arguments failed
[error]: expect Array, but got <*redis.BulkBytes>
    2   /root/.gvm/pkgsets/go1.5/global/src/github.com/CodisLabs/redis-port/pkg/redis/resp.go:163
            github.com/CodisLabs/redis-port/pkg/redis.AsArray
    1   /root/.gvm/pkgsets/go1.5/global/src/github.com/CodisLabs/redis-port/pkg/redis/handler.go:88
            github.com/CodisLabs/redis-port/pkg/redis.ParseArgs
    0   /root/.gvm/pkgsets/go1.5/global/src/github.com/CodisLabs/redis-port/cmd/sync.go:258
            main.(*cmdSync).SyncCommand.func2
        ... ...
[stack]:
    0   /root/.gvm/pkgsets/go1.5/global/src/github.com/CodisLabs/redis-port/cmd/sync.go:259
            main.(*cmdSync).SyncCommand.func2
        ... ...

猜测可能是某个key的的value太大导致的

目前有个list 类型的key value 大约有几千万条 - - ...

我该如何解决这个问题呢?

-ERR Unable to perform background save

2015/07/15 15:21:33 [panic]: invalid sync response, rsp = '-ERR Unable to perform background save
'
/home/go/src/github.com/wandoulabs/codis/ext/redis-port/cmd/utils.go:103
github.com/wandoulabs/codis/ext/redis-port/cmd.func·016

redis-port 迁移数据报错

2017/11/01 17:27:40 main.go:189: [INFO] set ncpu = 4, parallel = 16
2017/11/01 17:27:40 sync.go:56: [INFO] sync from '10.10.10.52:6679' to '10.10.10.55:8899'
2017/11/01 17:27:40 utils.go:139: [PANIC] invalid psync response, fullsync
[error]: bad resp CRLF end
6 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/pkg/redis/decoder.go:166
github.com/CodisLabs/redis-port/pkg/redis.(*decoder).decodeSingleLineBulkBytesArray
5 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/pkg/redis/decoder.go:85
github.com/CodisLabs/redis-port/pkg/redis.(*decoder).decodeResp
4 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/pkg/redis/decoder.go:28
github.com/CodisLabs/redis-port/pkg/redis.Decode
3 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/cmd/utils.go:137
main.sendPSyncFullsync
2 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/cmd/sync.go:115
main.(*cmdSync).SendPSyncCmd
1 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/cmd/sync.go:67
main.(*cmdSync).Main
0 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/cmd/main.go:199
main.main
... ...
[stack]:
3 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/cmd/utils.go:139
main.sendPSyncFullsync
2 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/cmd/sync.go:115
main.(*cmdSync).SendPSyncCmd
1 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/cmd/sync.go:67
main.(*cmdSync).Main
0 /home/zhouyi/go/src/github.com/CodisLabs/redis-port/cmd/main.go:199
main.main

迁移使用的命令是
./redis-port sync --psync --ncpu=4 -p 16 --from=10.10.10.52:6679 --filterdb=1 --target=10.10.10.55:8899 --sockfile=test.tmp --filesize=32gb
redis客户端是3.2版本的

使用redis-port sync 之后一直不停的刷新下面内容

您好:
我使用redis-port sync...命令来将redis的数据同步到codis中,但是一直出现下面内容在刷新,
请问:
1、sync是不是永远会执行下去?
2、“sync: +forward=12547 +nbypass=0 +nbytes=1588130”具体每个值是什么意思呢?比如+forward、+nbypass?
下面是刷屏内容:
2016/09/21 18:34:59 [INFO] sync: +forward=12547 +nbypass=0 +nbytes=1588130
2016/09/21 18:35:00 [INFO] sync: +forward=12121 +nbypass=0 +nbytes=1524604
2016/09/21 18:35:01 [INFO] sync: +forward=12721 +nbypass=0 +nbytes=1609632
2016/09/21 18:35:02 [INFO] sync: +forward=11180 +nbypass=0 +nbytes=1405242
2016/09/21 18:35:03 [INFO] sync: +forward=11215 +nbypass=0 +nbytes=1414128
2016/09/21 18:35:04 [INFO] sync: +forward=11677 +nbypass=0 +nbytes=1474217
2016/09/21 18:35:05 [INFO] sync: +forward=12818 +nbypass=0 +nbytes=1617235
2016/09/21 18:35:06 [INFO] sync: +forward=12489 +nbypass=0 +nbytes=1578302
2016/09/21 18:35:07 [INFO] sync: +forward=12898 +nbypass=0 +nbytes=1624360
2016/09/21 18:35:08 [INFO] sync: +forward=12558 +nbypass=0 +nbytes=1584386
2016/09/21 18:35:09 [INFO] sync: +forward=10635 +nbypass=0 +nbytes=1342169
2016/09/21 18:35:10 [INFO] sync: +forward=10354 +nbypass=0 +nbytes=1309103
2016/09/21 18:35:11 [INFO] sync: +forward=12291 +nbypass=0 +nbytes=1548088
2016/09/21 18:35:12 [INFO] sync: +forward=4331 +nbypass=0 +nbytes=549110
2016/09/21 18:35:13 [INFO] sync: +forward=11431 +nbypass=0 +nbytes=1441636
2016/09/21 18:35:14 [INFO] sync: +forward=10927 +nbypass=0 +nbytes=1376418
......

redis2.x迁移至4.x报错

源版本:Redis 2.8
目标版本:Redis 4.0

2018/08/19 19:53:50 [PANIC] parse rdb entry error
[error]: EOF
    10  /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:85
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).Read
    9   /usr/local/go/src/io/io.go:525
            io.(*teeReader).Read
    8   /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:83
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).Read
    7   /usr/local/go/src/io/io.go:309
            io.ReadAtLeast
    6   /usr/local/go/src/io/io.go:327
            io.ReadFull
    5   /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:290
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readFull
    4   /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:296
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).ReadBytes
    3   /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:216
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).ReadString
    2   /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/pkg/rdb/reader.go:169
            github.com/CodisLabs/redis-port/pkg/rdb.(*rdbReader).readObjectValue
    1   /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/pkg/rdb/loader.go:169
            github.com/CodisLabs/redis-port/pkg/rdb.(*Loader).NextBinEntry
    0   /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/cmd/utils.go:733
            main.newRDBLoader.func1
        ... ...
[stack]: 
    0   /home/pan.liangp/go/src/github.com/CodisLabs/redis-port/cmd/utils.go:734
            main.newRDBLoader.func1
        ... ...

redis-port执行抛错utils.go:106

go get 下载redis-port,make编译完成后,执行dump抛错,如下:
[root@ redis-port]# ./bin/redis-port dump -f 192.168.0.1:19000 -o data/IP-0-1.rdb
2015/12/11 17:40:08 [INFO] set ncpu = 1, parallel = 1
2015/12/11 17:40:08 [INFO] dump from '192.168.0.1:19000' to 'data/IP-0-1.rdb'
2015/12/11 17:40:08 [PANIC] read sync response = ''

[stack]:
0 /usr/local/go/gocode/src/github.com/wandoulabs/redis-port/cmd/utils.go:106
main.func路025
... ...
[root@ redis-port]#

redis-port在同步两个redis时,如何找到两个redis已经完全同步的时间点?

问题的场景是这样的:
利用redis-port同步redis1(master)和redis2(slave)。在完成了rdb的dump、传输、加载过程以及同步这段时间内的命令之后,锁住redis1(要求不能锁太久,在1秒之内),然后去做其他事情。做完之后,结束主从关系,并解锁redis1。

问题是:
1: 在锁redis1之前,如何判断主从已经达到同步状态?

2: 结束主从关系之前,如何确定主从已经达到同步状态?

关于数据的sync

[root@iZ23y2cpqz2Z redis-port]# ./bin/redis-port sync --from 10.168.22.140:6379 --filterdb=3 --target 10.251.241.246:19000
2016/07/28 12:39:17 [INFO] set ncpu = 2, parallel = 2
2016/07/28 12:39:17 [INFO] sync from '10.168.22.140:6379' to '10.251.241.246:19000'
2016/07/28 12:39:18 [INFO] +
2016/07/28 12:39:19 [INFO] -
2016/07/28 12:39:19 [INFO] rdb file = 37806381
2016/07/28 12:39:20 [INFO] total=37806381 - 9613493 [ 25%] entry=0 ignore=1472
2016/07/28 12:39:21 [INFO] total=37806381 - 20463960 [ 54%] entry=0 ignore=3010
2016/07/28 12:39:22 [INFO] total=37806381 - 34201728 [ 90%] entry=0 ignore=5363
2016/07/28 12:39:23 [INFO] total=37806381 - 37806381 [100%] entry=0 ignore=7258
2016/07/28 12:39:23 [INFO] sync rdb done
2016/07/28 12:39:24 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:39:25 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:26 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:27 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:28 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:39:29 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:30 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:31 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:32 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:33 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:34 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:35 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:36 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:37 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:38 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:39:39 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:40 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:41 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:42 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:43 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:44 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:45 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:46 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:47 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:48 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:39:49 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:50 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:51 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:52 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:53 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:54 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:55 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:56 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:57 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:58 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:39:59 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:40:00 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:01 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:02 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:03 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:04 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:05 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:06 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:07 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:08 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:09 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:40:10 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:11 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:12 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:13 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:14 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:15 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:16 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:17 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:18 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:19 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:40:20 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:21 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:22 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:23 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:24 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:25 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:26 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:27 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:28 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:29 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:40:30 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:31 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:32 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:33 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:34 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:35 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:36 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:37 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:38 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:39 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:40:40 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:41 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:42 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:43 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:44 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:45 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:46 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:47 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:48 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:49 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:40:50 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:51 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:52 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:53 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:54 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:55 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:56 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:57 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:58 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:40:59 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:41:00 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:01 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:02 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:03 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:04 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:05 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:06 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:07 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:08 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:09 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:41:10 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:11 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:12 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:13 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:14 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:15 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:16 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:17 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:18 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:19 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:41:20 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:21 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:22 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:23 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:24 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:25 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:26 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:27 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:28 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:29 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:41:30 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:31 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:32 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:33 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:34 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:35 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:36 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:37 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:38 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:39 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:41:40 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:41 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:42 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:43 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:44 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:45 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:46 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:47 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:48 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:49 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:41:50 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:51 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:52 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:53 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:54 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:55 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:56 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:57 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:58 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:41:59 [INFO] sync: +forward=1 +nbypass=0 +nbytes=14
2016/07/28 12:42:00 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:42:01 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:42:02 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:42:03 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:42:04 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:42:05 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
2016/07/28 12:42:06 [INFO] sync: +forward=0 +nbypass=0 +nbytes=0
......

以上的过程只要有sync rdb done 就算成功了吗,还是的一直等待那, 因为一直在forward; 刷屏;

我只能ctrl + c了.

restore 出错unknown command 'slotsrestore'

./bin/redis-port restore -i dump.rdb -t 127.0.0.1:6379 -n 8
2015/10/20 16:29:06 [INFO] set ncpu = 8, parallel = 8
2015/10/20 16:29:06 [INFO] restore from 'dump.rdb' to '127.0.0.1:6379'
2015/10/20 16:29:06 [PANIC] restore command error
[error]: ERR unknown command 'slotsrestore'
[stack]:
1 /usr/local/codis/src/github.com/wandoulabs/redis-port/cmd/utils.go:207
main.restoreRdbEntry
0 /usr/local/codis/src/github.com/wandoulabs/redis-port/cmd/restore.go:102
main.func·013
... ...

can not sync data from redis to codis

I use redis 2.8.21 of codis2.x, when i sync like this: ./redis-port sync --from=10.135.29.168:6381 --target=10.180.161.49:16379 --sockfile=redisport.tmp --filesize=2GB --psync , an error occurred :


2016/07/12 17:49:06 [PANIC] invalid sync response = '$EOF:915130babe4028703009b80a46956b6cac9adb54
', n = 0
[error]: strconv.ParseInt: parsing "EOF:915130babe4028703009b80a46956b6cac9adb54": invalid syntax

Then i capture tcp package, redis server return like below:
`*3
$5
## psync
$1
?
$2
-1
+FULLRESYNC 3c5def1ebe88c5fbc14c1b3a32dfed886dd5c81d 700308201
$EOF:8a4c50b7076680e1c6654b8901e22369946f24f9
REDIS0006....key-93794.AbCi.value-93794
[root@localhost conf.]# 2015/12/01 17:51:42 read tcp .10.58.65.240:52433->....64:2181: i/o timeout

I check the code,redis-port process by:

        if rsp[0] != '$' {
            log.Panicf("invalid sync response, rsp = '%s'", rsp)
        }
        n, err := strconv.Atoi(rsp[1 : len(rsp)-2])
        if err != nil || n <= 0 {
            log.PanicErrorf(err, "invalid sync response = '%s', n = %d", rsp, n)
        }

But redis process by:

retval = rdbSaveRioWithEOFMark(&slave_sockets,NULL);

int rdbSaveRioWithEOFMark(rio *rdb, int *error) {
    char eofmark[REDIS_EOF_MARK_SIZE];

    getRandomHexChars(eofmark,REDIS_EOF_MARK_SIZE);
    if (error) *error = 0;
    if (rioWrite(rdb,"$EOF:",5) == 0) goto werr;
    if (rioWrite(rdb,eofmark,REDIS_EOF_MARK_SIZE) == 0) goto werr;
    if (rioWrite(rdb,"\r\n",2) == 0) goto werr;
    if (rdbSaveRio(rdb,error) == REDIS_ERR) goto werr;
    if (rioWrite(rdb,eofmark,REDIS_EOF_MARK_SIZE) == 0) goto werr;
    return REDIS_OK;

how can solve this problem???

redis-port:panic: runtime error: invalid memory address or nil pointer dereference

使用redis-port同步数据从redis到codis,数据量有7G,400多万个key,在sync rdb阶段crash,感觉像是在建立到target连接读取返回值时hang住了?实验了多次,每次停在了不同的key上,应该和源数据没关系?下面是crash log。多谢
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x41caa9]

goroutine 7 [running]:
main.newRDBLoader.func1(0xc820075ec0, 0xc82001e8a0, 0xc820079e00)
/home/easemob/go/src/github.com/left2right/redis-port/cmd/utils.go:331 +0x639
created by main.newRDBLoader
/home/easemob/go/src/github.com/left2right/redis-port/cmd/utils.go:344 +0x68

goroutine 1 [select]:
main.(_cmdSync).SyncRDBFile(0xc820079e00, 0xc82001e8a0, 0x7fffd855a4e3, 0x26, 0x0, 0x0, 0x3b8a4c5f)
/home/easemob/go/src/github.com/left2right/redis-port/cmd/sync.go:227 +0xb2b
main.(_cmdSync).Main(0xc820079e00)
/home/easemob/go/src/github.com/left2right/redis-port/cmd/sync.go:91 +0x863
main.main()
/home/easemob/go/src/github.com/left2right/redis-port/cmd/main.go:377 +0x24c3

goroutine 17 [syscall, 6 minutes, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 8 [chan receive, 6 minutes]:
main.(_cmdSync).SyncRDBFile.func1(0xc820075f20, 0x7fffd855a4e3, 0x26, 0x0, 0x0, 0xc820075ec0, 0xc820079e00)
/home/easemob/go/src/github.com/left2right/redis-port/cmd/sync.go:222 +0x136
created by main.(_cmdSync).SyncRDBFile
/home/easemob/go/src/github.com/left2right/redis-port/cmd/sync.go:224 +0x110

goroutine 9 [IO wait]:
net.runtime_pollWait(0x2b88bf6e7388, 0x72, 0xc820076140)
/usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(_pollDesc).Wait(0xc82127eae0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc82127eae0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc82127ea80, 0xc82001d000, 0x1000, 0x1000, 0x0, 0x2b88bfd21028, 0xc820076140)
/usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(_conn).Read(0xc820030018, 0xc82001d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
bufio.(_Reader).fill(0xc8200740c0)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(_Reader).ReadSlice(0xc8200740c0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:328 +0x21a
github.com/garyburd/redigo/redis.(_conn).readLine(0xc8200ac000, 0x0, 0x0, 0x0, 0x0, 0x0)
/home/easemob/go/src/github.com/left2right/redis-port/Godeps/_workspace/src/github.com/garyburd/redigo/redis/conn.go:338 +0x5a
github.com/garyburd/redigo/redis.(_conn).readReply(0xc8200ac000, 0x0, 0x0, 0x0, 0x0)
/home/easemob/go/src/github.com/left2right/redis-port/Godeps/_workspace/src/github.com/garyburd/redigo/redis/conn.go:411 +0x57
github.com/garyburd/redigo/redis.(_conn).Do(0xc8200ac000, 0x69ff40, 0xc, 0xc824c0a7e0, 0x3, 0x3, 0x0, 0x0, 0x0, 0x0)
/home/easemob/go/src/github.com/left2right/redis-port/Godeps/_workspace/src/github.com/garyburd/redigo/redis/conn.go:559 +0x6b2
main.restoreRdbEntry(0x2b88bfd270e8, 0xc8200ac000, 0xc824bf0680)
/home/easemob/go/src/github.com/left2right/redis-port/cmd/utils.go:286 +0x1c6a
main.(_cmdSync).SyncRDBFile.func1.1(0xc8212056c0, 0x7fffd855a4e3, 0x26, 0x0, 0x0, 0xc820075ec0, 0xc820079e00)
/home/easemob/go/src/github.com/left2right/redis-port/cmd/sync.go:216 +0x377
created by main.(*cmdSync).SyncRDBFile.func1
/home/easemob/go/src/github.com/left2right/redis-port/cmd/sync.go:219 +0xe7

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.