redisjson/redisjson2

Rust based RedisJSON implementation - Deprecated and moved to RedisJSON/RedisJSON

RedisJSON

Overview

RedisJSON is a Redis module that implements ECMA-404 The JSON Data Interchange Standard as a native data type. It allows storing, updating and fetching JSON values from Redis keys (documents).

Primary features

• Full support of the JSON standard
• JSONPath syntax for selecting elements inside documents
• Documents are stored as binary data in a tree structure, allowing fast access to sub-elements
• Typed atomic operations for all JSON values types
• Secondary index support when combined with RediSearch

Quick start

docker run -p 6379:6379 --name redis-stack redis/redis-stack:latest

Documentation

Read the docs at https://redis.io/docs/latest/develop/data-types/json/

How do I Redis?

Learn for free at Redis University

Build faster with the Redis Launchpad

Try the Redis Cloud

Dive in developer tutorials

Join the Redis community

Work at Redis

Build

Make sure you have Rust installed: https://www.rust-lang.org/tools/install

Then, build as usual:

cargo build --release

When running the tests, you need to explicitly specify the test feature to disable use of the Redis memory allocator when testing:

cargo test

If you forget to do this, you'll see an error mentioning signal: 4, SIGILL: illegal instruction.

Run

Linux

redis-server --loadmodule ./target/release/librejson.so

Mac OS

redis-server --loadmodule ./target/release/librejson.dylib

Client libraries

Official clients

NRedisStack	Jedis	node-redis	redis-py
Redis.OM	Redis OM Spring	redis-om-node	redis-om

Community supported clients

Project	Language	License	Author	Stars	Package	Comment
Redisson	Java	Apache-2.0	Redisson		Maven
redis-modules-java	Java	Apache-2.0	Liming Deng @dengliming		maven
ioredis-rejson	Node.js	MIT	Felipe Schulz @schulzf		npm
go-rejson	Go	MIT	Nitish Malhotra @nitishm
rejonson	Go	Apache-2.0	Daniel Krom @KromDaniel
rueidis	Go	Apache-2.0	Rueian @rueian
NReJSON	.NET	MIT/Apache-2.0	Tommy Hanks @tombatron		nuget
phpredis-json	PHP	MIT	Rafa Campoy @averias		composer
redislabs-rejson	PHP	MIT	Mehmet Korkmaz @mkorkmaz		composer
rejson-rb	Ruby	MIT	Pavan Vachhani @vachhanihpavan		rubygems
rustis	Rust	MIT	Dahomey Technologies		crate	Documentation
coredis	Python	MIT	Ali-Akber Saifee @alisaifee		pypi	Documentation

Acknowledgments

RedisJSON is developed with <3 at Redis Labs.

RedisJSON is made possible only because of the existence of this amazing open source project:

• redis

License

RedisJSON is licensed under the Redis Source Available License 2.0 (RSALv2) or the Server Side Public License v1 (SSPLv1).

redisjson2's Issues

Build error[E0505]: cannot move out of `node` because it is borrowed

Hi,

when trying to build libredisjson.so I get the following issue:

$ cargo build --release
    Updating crates.io index
    Updating git repository `https://github.com/gkorland/jsonpath.git`
 Downloading crates ...
  Downloaded libc v0.2.66
  Downloaded log v0.4.8
  Downloaded env_logger v0.7.1
  Downloaded serde v1.0.104
  Downloaded array_tool v1.0.3
  Downloaded bson v0.14.0
  Downloaded redis-module v0.7.0
  Downloaded cfg-if v0.1.10
  Downloaded termcolor v1.1.0
  Downloaded atty v0.2.14
  Downloaded humantime v1.3.0
  Downloaded serde_derive v1.0.104
  Downloaded regex v1.3.4
  Downloaded serde_json v1.0.47
  Downloaded rand v0.7.3
  Downloaded num-traits v0.2.11
  Downloaded linked-hash-map v0.5.2
  Downloaded chrono v0.4.10
  Downloaded cc v1.0.50
  Downloaded md5 v0.6.1
  Downloaded quick-error v1.2.3
  Downloaded enum-primitive-derive v0.1.2
  Downloaded byteorder v1.3.4
  Downloaded time v0.1.42
  Downloaded bitflags v1.2.1
  Downloaded hex v0.3.2
  Downloaded ryu v1.0.2
  Downloaded proc-macro2 v1.0.8
  Downloaded rand_core v0.5.1
  Downloaded getrandom v0.1.14
  Downloaded thread_local v1.0.1
  Downloaded num-traits v0.1.43
  Downloaded quote v0.3.15
  Downloaded aho-corasick v0.7.8
  Downloaded rand_chacha v0.2.1
  Downloaded quote v1.0.2
  Downloaded indexmap v1.3.2
  Downloaded autocfg v1.0.0
  Downloaded num-integer v0.1.42
  Downloaded itoa v0.4.5
  Downloaded syn v1.0.14
  Downloaded memchr v2.3.0
  Downloaded redisearch_api v0.3.0
  Downloaded unicode-xid v0.2.0
  Downloaded lazy_static v1.4.0
  Downloaded regex-syntax v0.6.14
  Downloaded bindgen v0.52.0
  Downloaded lazycell v1.2.1
  Downloaded rustc-hash v1.1.0
  Downloaded shlex v0.1.1
  Downloaded bindgen v0.51.1
  Downloaded cmake v0.1.42
  Downloaded c2-chacha v0.2.3
  Downloaded cexpr v0.3.6
  Downloaded which v3.1.0
  Downloaded clap v2.33.0
  Downloaded syn v0.11.11
  Downloaded peeking_take_while v0.1.2
  Downloaded ppv-lite86 v0.2.6
  Downloaded env_logger v0.6.2
  Downloaded nom v4.2.3
  Downloaded clang-sys v0.28.1
  Downloaded ansi_term v0.11.0
  Downloaded vec_map v0.8.1
  Downloaded textwrap v0.11.0
  Downloaded unicode-xid v0.0.4
  Downloaded synom v0.11.3
  Downloaded unicode-width v0.1.7
  Downloaded strsim v0.8.0
  Downloaded glob v0.3.0
  Downloaded libloading v0.5.2
  Downloaded version_check v0.1.5
   Compiling memchr v2.3.0
   Compiling version_check v0.1.5
   Compiling cc v1.0.50
   Compiling libc v0.2.66
   Compiling bitflags v1.2.1
   Compiling proc-macro2 v1.0.8
   Compiling log v0.4.8
   Compiling glob v0.3.0
   Compiling lazy_static v1.4.0
   Compiling cfg-if v0.1.10
   Compiling autocfg v1.0.0
   Compiling unicode-width v0.1.7
   Compiling quick-error v1.2.3
   Compiling unicode-xid v0.2.0
   Compiling regex-syntax v0.6.14
   Compiling vec_map v0.8.1
   Compiling bindgen v0.52.0
   Compiling strsim v0.8.0
   Compiling ansi_term v0.11.0
   Compiling getrandom v0.1.14
   Compiling termcolor v1.1.0
   Compiling syn v1.0.14
   Compiling bindgen v0.51.1
   Compiling unicode-xid v0.0.4
   Compiling rustc-hash v1.1.0
   Compiling shlex v0.1.1
   Compiling peeking_take_while v0.1.2
   Compiling lazycell v1.2.1
   Compiling ryu v1.0.2
   Compiling ppv-lite86 v0.2.6
   Compiling quote v0.3.15
   Compiling serde v1.0.104
   Compiling byteorder v1.3.4
   Compiling itoa v0.4.5
   Compiling md5 v0.6.1
   Compiling linked-hash-map v0.5.2
   Compiling hex v0.3.2
   Compiling array_tool v1.0.3
   Compiling nom v4.2.3
   Compiling libloading v0.5.2
   Compiling cmake v0.1.42
   Compiling clang-sys v0.28.1
   Compiling thread_local v1.0.1
   Compiling textwrap v0.11.0
   Compiling humantime v1.3.0
   Compiling num-traits v0.2.11
   Compiling indexmap v1.3.2
   Compiling num-integer v0.1.42
   Compiling synom v0.11.3
   Compiling c2-chacha v0.2.3
   Compiling syn v0.11.11
   Compiling aho-corasick v0.7.8
   Compiling atty v0.2.14
   Compiling which v3.1.0
   Compiling time v0.1.42
   Compiling quote v1.0.2
   Compiling regex v1.3.4
   Compiling clap v2.33.0
   Compiling rand_core v0.5.1
   Compiling cexpr v0.3.6
   Compiling num-traits v0.1.43
   Compiling env_logger v0.7.1
   Compiling env_logger v0.6.2
   Compiling rand_chacha v0.2.1
   Compiling enum-primitive-derive v0.1.2
   Compiling chrono v0.4.10
   Compiling serde_derive v1.0.104
   Compiling rand v0.7.3
   Compiling redisearch_api v0.3.0
   Compiling serde_json v1.0.47
   Compiling bson v0.14.0
   Compiling jsonpath_lib v0.2.3 (https://github.com/gkorland/jsonpath.git?branch=patch-1#bd1975d6)
error[E0505]: cannot move out of `node` because it is borrowed
  --> /root/.cargo/git/checkouts/jsonpath-dd869ee3d7fc4db5/bd1975d/src/ffi/mod.rs:54:22
   |
53 |     let found = selector.compiled_path(&node).value(&json).select().unwrap();
   |                                         ---- borrow of `node` occurs here
54 |     std::mem::forget(node);
   |                      ^^^^ move out of `node` occurs here
error: aborting due to previous error
For more information about this error, try `rustc --explain E0505`.
error: Could not compile `jsonpath_lib`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Any ideas what to do about it? Am I missing some dependecies? Speaking of dependencies it would be nice to have a list of them somewhere. Maybe in README.md.

Should be able query docs with missing paths in index

127.0.0.1:6379> json.index add inx first $.first
OK
127.0.0.1:6379> json.index add inx llast $.last
OK
127.0.0.1:6379> json.set user1 . '{"first":"Joe"}' INDEX inx
(error) ERR path does not exist
127.0.0.1:6379> json.set user1 . '{"first":"Joe", "last": "Smith"}' INDEX inx
OK

Add support for MGET

https://oss.redislabs.com/rejson/commands/#jsonmget

Docker image in Hub

Does it make sense to tag the 2nd version in the rejson repository?

Or it's too early to use it?

Why license is not BSD?

Digital Ocean just refused to install this module into their redis-as-a-service because it's not BSD licensed. Redis is BSD licensed, why ReJson doesn't have BSD license?

Failed build --release

Hello, I downloaded the code of the master branch and when executing the command to compile (cargo build --release) I get the following error.

What should I do to compile the code and get the module for Redis?

Note: I am using the ubuntu subsystem in windows

Thanks.

JSON.ARRINDEX where stop is 0 or -1 should include the last element

127.0.0.1:6379> "JSON.SET" "arr" "." "[0, 1, 2, 3, 4]"
OK
127.0.0.1:6379> "JSON.ARRINDEX" "arr" "." "3" "2" "-1"
(error) invalid digit found in string
127.0.0.1:6379> "JSON.ARRINDEX" "arr" "." "3" "2" "0"
(integer) -1

Both calls should return

(integer) 3

Add support for STRLEN

https://oss.redislabs.com/rejson/commands/#jsonstrlen

Complete support for SET

JSON.SET <key> <path> <json>   [NX | XX]

https://oss.redislabs.com/rejson/commands/#jsonset

JSON.GET with multi path should return original path for backward

JSON.GET obj bTrue str

Returns

{
  "$.bTrue":true,
 "$.str":"string"
}

Should have return

{
  "bTrue":true,
  "str":"string"
}

JSON.TYPE on Integer should return `Integer`

127.0.0.1:6379> JSON.SET 1 . 1
127.0.0.1:6379> JSON.TYPE 1 .
"number"

Should return

"integer"

strappend should remove surrounding ""

127.0.0.1:6379> JSON.SET str . '"foo"'
OK
(5.49s)
127.0.0.1:6379> JSON.Strlen str . 
(integer) 3
127.0.0.1:6379> JSON.STRAPPEND str . '"bar"'
(integer) 8

Should have return

(integer) 6

Define a testing methodology

Should include:

• Unit tests (inside the source files, per the Rust convention)
• Integration/component tests that test against Redis with the module.

Add support for ARRAPPEND

https://oss.redislabs.com/rejson/commands/#jsonarrappend

Index existing data when a new Index is created

Add support for NUMINCRBY

https://oss.redislabs.com/rejson/commands/#jsonnumincrby

Add support for query by Index

Current MGET

JSON.MGET <key> [key ...] <path>

Future should support FILTER

JSON.FIND {query} <path>

e.g.

JSON.FIND "@country:korea @tech.engine:(diesel|hybrid) @class[*]:suv" $.brand

Or

JSON.FIND "@$.country:korea @$.tech.engine:(diesel|hybrid) @$.class[*]:suv" $.brand

Add support for NUMMULTBY

https://oss.redislabs.com/rejson/commands/#jsonnummultby

Add support of JsonPath filter (script) expression to JSON.GET

Hi, I tried to use JsonPath filter expression and got Invalid argument(s) error :

127.0.0.1:6379> JSON.GET d [10]
"{\"code\":\"AAL\",\"coordinates\":{\"lat\":57.028811,\"lon\":9.917771},\"country_code\":\"DK\",\"name\":\"\xe3\x82\xaa\xe3\x83\xbc\xe3\x83\xab\xe3\x83\x9c\xe3\x83\xbc\",\"name_translations\":{\"en\":\"Aalborg\"},\"time_zone\":\"Europe/Copenhagen\"}"
127.0.0.1:6379> JSON.GET d .[?(@.code=='AAL')]
Invalid argument(s)
127.0.0.1:6379> JSON.GET d .[(@.code=='AAL')]
Invalid argument(s)

Seems like filter expressions is not supported yet, right ?
If it's supported - please provide example. Thanks a lot.

Please sign the CLA so we can merge your commit upstream

Dear contributors -- @VadimKulagin and @simonprickett

We are in the process of releasing the new Rust-based RedisJSON. As part of this, we are merging the code from this repo to the https://github.com/RedisJSON/RedisJSON repo.

You both have contributed patches:

• https://github.com/RedisJSON/RedisJSON2/commits?author=VadimKulagin
• https://github.com/RedisJSON/RedisJSON2/commits?author=simonprickett

All contributions to that repo require the (fairly standard) process of signing our CLA:
https://cla-assistant.io/RedisJSON/RedisJSON

We would like to keep your contributions, under your respective names.
If you would be kind enough to sign the above CLA, we can merge your patches as they currently are.

Thanks for the cooperation,
Gavrie Philipson at Redis Labs.

Crash on wrong JSON.SET path syntax

127.0.0.1:6379> JSON.SET "DATA" . {}
OK
127.0.0.1:6379> JSON.SET "DATA" $a 12
Could not connect to Redis at 127.0.0.1:6379: Connection refused

Add support for TYPE

https://oss.redislabs.com/rejson/commands/#jsontype

JSON.QGET doesn't work on non-String fields

Hi,

thanks for providing the community with the RedisJSON2 extension!

TL;DR

The indexing (or querying) mechanism of RedisJSON2 does not seem to work for non-string fields.

Could this be fixed, so that JSON.QGET also allows queries involving non-string fields?

Detailed description (including reproducer) follows

I'm creating an index users as follows and insert two data sets:

1573724213.769257 [0 127.0.0.1:58322] "SELECT" "7"
1573724213.769399 [7 127.0.0.1:58322] "FLUSHDB"

1573724584.246502 [7 127.0.0.1:58322] "JSON.INDEX" "ADD" "users" "first" "$.first"
1573724584.246800 [7 127.0.0.1:58322] "JSON.INDEX" "ADD" "users" "last" "$.last"
1573724584.247056 [7 127.0.0.1:58322] "JSON.INDEX" "ADD" "users" "age" "$.age"

1573724648.064295 [7 127.0.0.1:60454] "JSON.SET" "user1" "." "{\"first\": \"Joe\", \"last\": \"Smith\"}" "INDEX" "users"
1573724651.026177 [7 127.0.0.1:60454] "JSON.SET" "user2" "." "{\"first\": \"Kevin\", \"last\": \"Smith\", \"age\": 33}" "INDEX" "users"

Next I'm executing a query asking for all users whose last names start with "smi":

1573724379.240656 [7 127.0.0.1:58322] "JSON.QGET" "users" "@last:smi*"

The query returns all users as expected:

{"user1":[{"first":"Joe","last":"Smith"}],"user2":[{"first":"Kevin","last":"Smith","age":33}]}

However when asking for users with an age of 33:

1573724848.397281 [7 127.0.0.1:60454] "JSON.QGET" "users" "@age:33"

Nothing is returned:

{}

If I insert a third user with an age specified as string (note the difference to the user2 insertion):

1573725060.306817 [7 127.0.0.1:60454] "JSON.SET" "user3" "." "{\"first\": \"Some\", \"last\": \"Dude\", \"age\": \"33\"}" "INDEX" "users"

And rerun the query:

1573725213.414359 [7 127.0.0.1:60454] "JSON.QGET" "users" "@age:33"

The data set for user3 is returned.

{"user3":[{"first":"Some","last":"Dude","age":"33"}]}

This observation leads to the conclusion that the indexing (or querying) mechanism of RedisJSON2 does not work for non-string fields.

Could this be fixed, so that JSON.QGET also allows queries involving non-string fields, like the one I created for user2?

Following you'll find my reproducer script that was hacked together in Python. I used redis-cli monitor to monitor the actual commands hitting redis as shown above.

import redis
import json


redis_host = "localhost"
redis_db = 7
r = redis.StrictRedis(host=redis_host, db=redis_db)

r.flushdb()

# %%

index = 'users'

try:
    r.execute_command('JSON.INDEX', 'ADD', index, 'first', '$.first')
except:
    print("Index probably exists")
try:
    r.execute_command('JSON.INDEX', 'ADD', index, 'last', '$.last')
except:
    print("Index probably exists")
try:
    r.execute_command('JSON.INDEX', 'ADD', index, 'age', '$.age')
except:
    print("Index probably exists")

# %%

data_1 = {
    'first': 'Joe',
    'last': 'Smith'
}

r.execute_command('JSON.SET', 'user1', '.', json.dumps(data_1), 'INDEX', index)

# %%

data_2 = {
    'first': 'Kevin',
    'last': 'Smith',
    'age': 33
}

r.execute_command('JSON.SET', 'user2', '.', json.dumps(data_2), 'INDEX', index)

# %%

print(r.execute_command('JSON.QGET', index, '@last:smi*'))

# %%

print(r.execute_command('JSON.QGET', index, '@age:33'))

# %%

data_3 = {
    'first': 'Some',
    'last': 'Dude',
    'age': '33'
}

r.execute_command('JSON.SET', 'user3', '.', json.dumps(data_3), 'INDEX', index)

# %%

print(r.execute_command('JSON.QGET', index, '@age:33'))

Crash on json.get

When using $ symbol redis crashed (without it works fine)

127.0.0.1:6379> json.set user2 . '[{"code":"1"}, {"code":"2"}]'
OK
127.0.0.1:6379> json.get user2 ".[?(@.code==\"2\")]"
"{\"code\":\"2\"}"
127.0.0.1:6379> json.get user2 "$.[?(@.code==\"2\")]"
Error: Connection reset by peer

=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 10 Feb 2020 10:20:13.337 # Redis 5.0.7 crashed by signal: 11
1:M 10 Feb 2020 10:20:13.337 # Crashed running the instruction at: 0x7f90f4a1c8fc
1:M 10 Feb 2020 10:20:13.337 # Accessing address: (nil)
1:M 10 Feb 2020 10:20:13.337 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/usr/lib/redis/modules/libredisjson.so(+0xa98fc)[0x7f90f4a1c8fc]

Backtrace:
redis-server *:6379(logStackTrace+0x32)[0x555f4530c572]
redis-server *:6379(sigsegvHandler+0x9e)[0x555f4530cc4e]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f90f576b730]
/usr/lib/redis/modules/libredisjson.so(+0xa98fc)[0x7f90f4a1c8fc]
/usr/lib/redis/modules/libredisjson.so(+0xa9d85)[0x7f90f4a1cd85]
/usr/lib/redis/modules/libredisjson.so(+0x9e3e7)[0x7f90f4a113e7]
/usr/lib/redis/modules/libredisjson.so(+0x8fc5c)[0x7f90f4a02c5c]
/usr/lib/redis/modules/libredisjson.so(+0x8fcb8)[0x7f90f4a02cb8]
/usr/lib/redis/modules/libredisjson.so(+0x8fab3)[0x7f90f4a02ab3]
/usr/lib/redis/modules/libredisjson.so(+0x93db2)[0x7f90f4a06db2]
/usr/lib/redis/modules/libredisjson.so(+0x8cc45)[0x7f90f49ffc45]
/usr/lib/redis/modules/libredisjson.so(+0x8cc91)[0x7f90f49ffc91]
/usr/lib/redis/modules/libredisjson.so(+0x8f7b0)[0x7f90f4a027b0]
/usr/lib/redis/modules/libredisjson.so(+0x8f8dc)[0x7f90f4a028dc]
/usr/lib/redis/modules/libredisjson.so(+0x9d2bb)[0x7f90f4a102bb]
/usr/lib/redis/modules/libredisjson.so(+0x7b5ce)[0x7f90f49ee5ce]
/usr/lib/redis/modules/libredisjson.so(+0x7b465)[0x7f90f49ee465]
/usr/lib/redis/modules/libredisjson.so(+0x35dce)[0x7f90f49a8dce]
redis-server *:6379(RedisModuleCommandDispatcher+0x54)[0x555f45338d14]
redis-server *:6379(call+0x9b)[0x555f452c835b]
redis-server *:6379(processCommand+0x51e)[0x555f452c8c1e]
redis-server *:6379(processInputBuffer+0x171)[0x555f452d8e61]
redis-server *:6379(aeProcessEvents+0x101)[0x555f452c2311]
redis-server *:6379(aeMain+0x2b)[0x555f452c271b]
redis-server *:6379(main+0x4ca)[0x555f452bf57a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f90f55bc09b]
redis-server *:6379(_start+0x2a)[0x555f452bf7da]

ref #79

json.num* operations should treat integers as such

127.0.0.1:6379> JSON.set test . '{ "foo": 0, "bar": "baz" }'
OK
127.0.0.1:6379> JSON.NUMINCRBY test .foo 1
"1.0"

Should have return

"1"

JSON.ARRPOP should return the poped object and not the leftover

Wrong:

127.0.0.1:6379> json.set arr . '[0,1,2,3,4]'
OK
127.0.0.1:6379> JSON.ARRPOP arr . 4
"[0,1,2,3]"

Right:

127.0.0.1:6379> json.set arr . '[0,1,2,3,4]'
OK
127.0.0.1:6379> JSON.ARRPOP arr . 4
"4"

Add support for Secondary index

Using RediSearch API we should be able to add support for a Secondary index

JSON.CREATEINDEX path 

Full API should probably support more index properties e.g.

JSON.CREATEINDEX path [ALIAS alias] [TEXT [NOSTEM] [WEIGHT {weight}] [PHONETIC {matcher}] | NUMERIC | GEO | TAG [SEPARATOR {sep}] ] [SORTABLE][NOINDEX] 

Add support for DEL

https://oss.redislabs.com/rejson/commands/#jsondel

Add support for STRAPPEND

https://oss.redislabs.com/rejson/commands/#jsonstrappend

Missing container image in registry

Tried to pull the image down tonight, appears that it is not available?

docker pull redislabs/redisjson2:latest
Error response from daemon: manifest for redislabs/redisjson2:latest not found

redislabs/redisjson2:edge appears to work. Maybe just a documentation or tagging issue?