jwhitbeck/java-rdb-parser

A simple Redis RDB file parser for Java

java-rdb-parser's Issues

signed integer -1 parsing error

hi !
I would like to inform you that while parsing dump.rdb file, I found the method written below seems to have a problem in case of signed data.

private byte[] readInteger8Bits() throws IOException { return String.valueOf(readByte()).getBytes(ASCII); }

private int readByte() throws IOException { if (!buf.hasRemaining()) { fillBuffer(); } return buf.get() & 0xff; }

The problem that I recognized was when I'm using this method, the figure -1 comes out wrong result 255 because the upfront bit cannot recognize it 'signed'.

To solve this problem, I think you should remove the '& 0xff' Operation Code in readByte method.

On top of that, I would like you to confirm whether readInteger16Bits, readInteger32Bits methods also have sames issues above.

I will be appreciated for your feedback as far as you can reach. :)

parse rdb serialized by protocol buffer

Hi, I try to use this tool to parse rdb whose value is serialized by protocol buffer. But it fail. What should I do to deal with the problem. Code is as below:

public static void parseIdMappingRdbFile(String rdbFilePath) {
        File rdbFile = new File(rdbFilePath);
        RdbParser parser = null;
        try {
            parser = new RdbParser(rdbFile);
        } catch (IOException e) {
            logger.error("fail to parse rdb file[{}]", rdbFile);
            System.exit(-1);
        }
        Entry parsedRdbEntry = null;
        while (true) {
            try {
                parsedRdbEntry = parser.readNext();
                if (null == parsedRdbEntry) {
                    break;
                }
            } catch (IOException e) {
                e.printStackTrace();
                continue;
            }
            if (parsedRdbEntry.getType().equals(KEY_VALUE_PAIR)) {
                KeyValuePair kvp = (KeyValuePair)parsedRdbEntry;
                List<byte[]> values = kvp.getValues();
                if (null == values || values.size() == 0) {
                    continue;
                }
                for (byte[] value : values) {
                    try {
                         // ** here the problem happens: cannot parse from value from rdb file **
                         IdMappingProto.IdMapping idMapping = IdMappingProto.IdMapping.parseFrom(value);
                    } catch (InvalidProtocolBufferException e) {
                        logger.error("[{}]:fail to deserialize class of IdMapping", cnt);  
                        continue; 
                    }
                }
            }
        }
}

Support Version 8

Let's support the dump file format version 8. It's related to #3 .

ArrayIndexOutOfBoundsException

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at net.whitbeck.rdbparser.ZipList.get(ZipList.java:60)
	at net.whitbeck.rdbparser.QuickList.get(QuickList.java:30)
	at net.whitbeck.rdbparser.KeyValuePair.getValues(KeyValuePair.java:173)
	at RdbParserTest.main(RdbParserTest.java:30)

Report the bytes consumed

We use this tool to analyse RDB file and output the records to our Analytics Database, but we need to know how many bytes we scanned.

So can we add a new method to get the scanned bytes when parser called

Can not support RedisBloom Module

hi, I use RedisBloom module in redis , the RdbParser now can not parse the bloom filter which in rdb file , is there any plan to support RedisBloom Module

My environment info below:
redis version: 6.2.5
RedisBloom version: 2.2.6

Cannot convert key in key-values redis

Here is the result . It's not convert correctly.

Support Redis4.0 Binary Double

private KeyValuePair readEntry(byte[] ts, int valueType) throws IOException {
add:
case 5:
return readSet2(ts, key);

// REDIS_RDB_TYPE_ZSET_2 = 5 # ZSET version 2 with doubles stored in binary.
private KeyValuePair readSet2(byte[] ts, byte[] key) throws IOException {
long len = readLength();
if (len > Integer.MAX_VALUE) {
throw new IllegalArgumentException("Sets with more than " + Integer.MAX_VALUE + " elements are not supported.");
}
int size = (int) len;
List<byte[]> valueScoresPairs = new ArrayList<byte[]>(2 * size);
for (int i = 0; i < size; ++i) {
byte[] valueBuf = readStringEncoded2();
valueScoresPairs.add(valueBuf);
byte[] valueDBuf = readLongString();
valueScoresPairs.add(valueDBuf);
}
return new KeyValuePair(ValueType.SET, ts, key, valueScoresPairs);
}

private byte[] readLongString() throws IOException {
return readBytes(8);
}

if (kvp.getValueType() == ValueType.SET) {
// 2. zadd
for (Iterator<byte[]> i = kvp.getValues().iterator(); i.hasNext();) {
String uuid = str(i.next());
long score = strLong(i.next());
zadd(key, uuid, score, expTime);
}
}

static long strLong(byte[] readBuffer) throws Exception {
long value = (long) toDouble(readBuffer);
return value;
}

public static double toDouble(byte[] bytes) {
return ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).getDouble();
}

UnsupportedOperationException: Unknown value type: 248

Hello,

I am trying the parser and I am seeing "Unknown value type: 248". I wonder if you have any suggestions.
The issue seems to happen when I set an expire for the HSET. I created minimal code to reproduce the issue:

127.0.0.1:6379> INFO SERVER
# Server
redis_version:6.0.10
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:d4a632e656c1d2c6
redis_mode:standalone

Create a HSET with expire:

127.0.0.1:6379> FLUSHDB
OK
127.0.0.1:6379> HSET myhash values xxxx
(integer) 1
127.0.0.1:6379> EXPIRE myhash 900000
(integer) 1
127.0.0.1:6379> SAVE
OK

Error:

Exception in thread "main" java.lang.UnsupportedOperationException: Unknown value type: 248
	at net.whitbeck.rdbparser.RdbParser.readEntry(RdbParser.java:402)
	at net.whitbeck.rdbparser.RdbParser.readEntryMillis(RdbParser.java:366)
	at net.whitbeck.rdbparser.RdbParser.readNext(RdbParser.java:196)
	at debug.Debug.printRdbFile(Debug.java:15)
	at debug.Debug.main(Debug.java:9)

Remove the expire:

127.0.0.1:6379> PERSIST myhash
(integer) 1
127.0.0.1:6379> SAVE
OK

Now it works:

Processing DB: 0
------------
Key value pair
Key: myhash
Value type: HASHMAP_AS_ZIPLIST
Values: values xxxx 
------------
End of file. Checksum: c0e7e68c33cf9434
------------

I'd appreciate any suggestions. thanks! (hopefully I am not doing something dumb)

parse dump.rdb don't parse all keys

hi,i use your great program,but parse dump.rdb don't parse all keys
my redis version 3.2.1

Support new RDB format introduced by redis 3.2

According to the Redis 3.2 release notes, the RDB format has changed. This project should support it.

Why for some value of integers, return String representation other then bytes directly.

In ZipList.java, ZipList::get(), why for integer encodings, it does:

list.add(String.valueOf(v).getBytes(ASCII));

Can we just return bytes? Bytes is more consistent in all other parts of codes.

How do I do it ? jedis.dump("name")

jedis.set("name","redis");
byte[] b = jedis.dump("name");
String name = parseDump("name")//How do I do it???