antirez/resp3

RESP protocol V3 repository. Contains the specification, and other related resource

This is the repository for the version 3 of the RESP protocol. RESP (REdis Serialization Protocol) is the protocol used in the Redis database, however the protocol is designed to be used by other projects. With the version 3 of the protocol, currently a work in progress design, the protocol aims to become even more generally useful to other systems that want to implement a protocol which is simple, efficient, and with a very large landscape of client libraries implementations.

Here you can find the current RESP3 specification.

In the future this repository may add additional resources, such as example implementations of RESP or other resources.

resp3's Issues

Side-Channel/Control Plane Information

Hey Redis Team!

I've looked through the RESP3 spec, it looks great!

One thing that sprung to mind is passing metadata in the request/response cycle. In MySQL, for example, we tend to pass a comment in each query with where the query came from (path, request id, ..). This is very helpful in the slow query log, and for instrumenting with e.g. BPF:

Account Load (0.3ms)  SELECT `accounts`.* FROM `accounts` 
WHERE `accounts`.`queenbee_id` = 1234567890 
LIMIT 1 
/*application:xx,controller:project_imports,action:show,request_id:x, ..*/

In the response cycle, it's possible to attach metadata as well (continuing to use MySQL as an example). We anticipate we'll soon through a plugin we will pass back information such as the CPU, Network and IO cost of a query, as well as the utilization of these resources. This will help throttle further up the stack, e.g. shed load the load-balancers or prioritize queries. We think this could be useful for Redis too, longer-term.

As far as I can understand this is not something RESP3 in its current form will support. The closest I can see is Push, which is out-of-band data, that will not be tied to a query.

Curious what you think, or whether I missed something! 👀

Change attribute type example

The current first example in attribute type is the following

|1<CR><LF>
    +key-popularity<CR><LF>
    %2
        :a
        ,0.1923
        :b
        ,0.0012
*2<CR><LF>
    :2039123
    :9543892

Ignoring the problems pointed out in #3, I think a better response and more clear documentation would be the following

*2<CR><LF>
    |1<CR><LF>
        %2<CR><LF>
        +key<CR><LF>
        $1<CR><LF>
        a<CR><LF>
        +popularity<CR><LF>
        ,0.1923<CR><LF>
    :2039123<CR><LF>
    |1<CR><LF>
        %2<CR><LF>
        +key<CR><LF>
        $1<CR><LF>
        b<CR><LF>
        +popularity<CR><LF>
        ,0.0012<CR><LF>
    :9543892<CR><LF>

This is actually more consistent with the other example where a single element has an attribute.

*3<CR><LF>
    :1<CR><LF>
    :2<CR><LF>
    |1<CR><LF>
        +ttl
        :3600
    :3<CR><LF>

I think the name and description is also confusing. I suggest calling it "metadata" and the description could change to something like this

Metadata type contains metadata preceding the value to be returned in the form of a dictionary. It works like a map type with auxiliary information and it is followed by the actual value to be returned.

Attribute type

I just went ahead and implemented RESP3 for the Node.js redis parser and I struggle to see the how this can be delivered to the user in a useful way.

It is not clear if the attribute alone should be delivered to the user or if it should be delivered in combination with the actual corresponding data (which IMHO would make sense). The latter however requires the parsing logic to become more complex in cases for aggregated data types such as arrays, maps and sets.

It is also not clear when exactly the data should be delivered as in: immediately after the attribute is parsed or after the top level is done parsing? (This is again about aggregated data types)

Would it not be better to just provide the very same information as push information that just contains the actual data on top of the attribute? It's of course extra data to parse but it would simplify the parser logic. Right now the attributes somewhat seem out of line for me compared to the rest of the spec.

Text type?

I wonder what you think of adding a dedicated text type, which would be identical to the bulk-string, but would indicate UTF8-encoded unicode text as opposed to an arbitrary blob of bytes?

The difference to the client would be equivalent to how Python 3 differentiates str (for unicode) and bytes (for blobs). I think msgpack has something similar.

Or perhaps this is the intended use-case for the "verbatim string" type?

Improving aggregated data types for performance

For me the most challenging type while implementing RESP in JS was arrays. The input comes as an event with partial data without knowing when the array ends. It would be great to know the actual size of the aggregated data up front similar to bulk strings so it's possible to wait with the parsing until all necessary data has been received. That requires Redis to actually calculate the size up front but the receiver could implement it more efficiently, especially for nested aggregated types. It's off course a trade-off but I believe it could improve the overall performance for these types quite a bit (mainly in case of many entries).

I am curious about your thoughts about this @antirez.

Is map type unordered?

Spec suggests encoding json as an example and json is unordered, however it doesn't explicitly states this.

Ordered hash maps would complicate implementation for some languages, like go(map) or python(dict).

Reconsider format for streamed types

Streamed types are modeled with an unknown length. However, the individual chunk lengths are known. Defining an EOF marker and stream-parsing the EOF marker introduces a certain degree of complexity to clients.

Why we don't stick to a pattern such as partial length-prefixed chunks? Each chunk is prefixed with a length indicator (similar to strings) and a negative length value would serve for the EOF marker.

?10<CR><LF>
<10 bytes of data><CR><LF>
?400<CR><LF>
<400 bytes of data><CR><LF>
?-1<CR><LF>    <----- EOF marker

(using ? here as there was no other free character as a prefix for a stream response type).

Justifications for not using existing MessagePack / BSON, etc.

I think your section about alternatives is open to interpretation and maybe the reasoning can be improved.

The first and most important is the fact that such serialization protocols are not specifically designed for a request-response server-client chat, so clients and servers would be required to agree on an additional protocol on top of the underlying serialization protocol.

At least with MessagePack, you can stream binary messages very easily. So, request/response is pretty trivial to implement.

A different problem is the fact that such serialization protocols are more complex than RESP, so client libraries would have to rely on a separated library implementing the serialization protocol.

In my mind, this is both a pro and a con. Obviously, the reason you mention makes sense. However, in every library I've seen, some kind of protocol parser must be implemented. Honestly, I'd rather just use a 3rd party implementation like MessagePack which is proven to be fast/memory efficient, than have every library roll their own.

Finally certain features like transferring large strings with initially unknown length, or streaming of arrays, two features that RESP3 supports and that this specification describes, are not easy to found among existing serialization protocols.

While this might be more tricky to implement, there is certainly no reason why you can't split such replies up into continuation messages. Given that you want to have features like client key cache eviction, streaming huge amounts of data is often a bad idea. It also introduces memory usage issues as client has no choice but to read all data to reconstruct the reply, rather than handling discrete chunks.

Not saying I don't agree with the points, I just think they don't present a clear cut reason not to use MessagePack, BSON, etc.

Honestly, from my POV, I wish you just used some existing format, rather than inventing your own. Even if you needed to add some semantics on top of it, it would be easier for client implementations IMHO. So, best to make a really strong case for RESP3.

Blob helloworld is incorrect

The first protocol example in the spec is:
"$11\r\nhelloworld\r\n"
This is incorrect.

Either of these would be correct:
"$10\r\nhelloworld\r\n"
"$11\r\nhello world\r\n"

Attributes and Push Types

I'm writing a low-level Redis client and I'm now at the stage where I'm thinking about implementing Pub/Sub support and I found an annoying-to-solve problem with regards to the push type and attributes.

Without considering attributes, since push types can be sent in-between normal replies from Redis, the parser must account for the possibility of finding something that is not the expected reply, but rather a push type. This is easy to solve as it just needs to check the type tag (> vs any of the others) and react accordingly.

Now, if we add attributes to the mix, this becomes a bit more complex.
My understanding of the RESP3 spec is that attributes are also valid for push types. If that's the case, then when the parser encounters a | it won't know if what lies ahead is a reply or a push. This is solvable by doing some look-ahead, but since this is the only case where such a thing is needed, and since I believe ease of implementation is one of the main goals, I'd recommend making some minor changes to avoid this problem.

To re-state the problem in more concrete terms, consider the following examples where the parser is being invoked by the user calling client.send(i64, "INCR", "key").

First case:

>4<CR><LF>
+pubsub<CR><LF>
+message<CR><LF>
+somechannel<CR><LF>
+this is the message<CR><LF>
:123<CR><LF>

Second case:

:123<CR><LF>
>4<CR><LF>
+pubsub<CR><LF>
+message<CR><LF>
+somechannel<CR><LF>
+this is the message<CR><LF>

As you can see in these two cases reading the first tag is enough to disambiguate. In the first case the parser will know that it needs to delegate the parsing of the first message accordingly and then retry in order to get the reply to INCR.

Third case:

|1<CR><LF>
    +channel-popularity<CR><LF>
    :100<CR><LF>
>4<CR><LF>
+pubsub<CR><LF>
+message<CR><LF>
+somechannel<CR><LF>
+this is the message<CR><LF>
:123<CR><LF>

Fourth case:

|1<CR><LF>
    +key-popularity<CR><LF>
    :5<CR><LF>
:123<CR><LF>
>4<CR><LF>
+pubsub<CR><LF>
+message<CR><LF>
+somechannel<CR><LF>
+this is the message<CR><LF>

Note how in the third and fourth case it's impossible to distinguish between the two just by looking at the first tag found in the stream. As stated earlier, this is not an insurmountable problem, but it does add an irregularity in the protocol by being the only case where reading the next type tag is not enough to parse a message correctly.

Not allowing top-level attributes for push messages, or using a different tag would be enough for ensuring we can disambiguate each case just by looking at the next byte in the stream.

Looking forward to hear some feedback about this.

Thanks,
Loris

Optional checksum support

Hello,

What is the feeling towards adding optional end-to-end checksum support to RESP3? The checksum in TCP is quite poor, and will often fail to detect corrupted network packets. This is not theoretical, and there's published papers that demonstrate it in real-world scenarios.

https://stackoverflow.com/questions/3830206/can-a-tcp-checksum-fail-to-detect-an-error-if-yes-how-is-this-dealt-with
https://dl.acm.org/citation.cfm?id=347561&dl=GUIDE&coll=GUIDE

After an analysis we conclude that the checksum will fail to detect errors for roughly 1 in 16 million to 10 billion packets.
...
Even so, the highly non-random distribution of errors strongly suggests some applications should employ application-level checksums or equivalents.

If a client sends "SET key foo", but the server receives "SET key f0o" due to network corruption, there's no way to detect anything went wrong. Similarly, when the server sends some data, there's no way to detect client-side if the contents were somehow corrupted.

It could work like this: On HELLO, the client could also request that all top-level requests and replies are prepended with a particular fixed-size checksum:

HELLO 3 CHECKSUM xxhash AUTH default blabla
->   <4 checksum bytes> Reply
<4 checksum bytes> SET abc 123
->  <4 checksum bytes>+ OK
<4 checksum bytes> HGET 123 abc
-> <4 checksum bytes><data in abc field of hash 123>

Clients not supporting checksums would still work - they'd simply not request them. When a checksum error is detected, either on the client side or the server side, we'd simply shut down the connection and treat it as a protocol parsing error, so the client will simply retry, just as if this was a transient network instability.

What do you think?

Syntax highlighting in the spec snippets

For someone who's not an expert in Redis, <CR><LF> and \r\n bits are quite verbose and distracting, while ; and , are probably very important (?) but easy to miss.

I wish the spec used syntax highlighting to help focus the reader on the important parts of the "code" snippets.

Consider line break change

Since RESP3 is going to be backwards incompatible, wouldn't this be a good time to switch from <CR><LF> to just <LF>? In the service of making it an even more compact protocol, it would still be basically as readable. Maybe there's a very good reason to use <CR><LF> in RESP in general that I'm not aware of though.

Different authentication mechanisms?

Has there been any consideration for extending the HELLO message to include another arguments such as different types of authentication that require more or fewer arguments that could be used as a module hook in the future.

e.g.
HELLO 1 [AUTH | IAM ]

There are probably other ways to hook into authentication in the future, but this seems like one of the ways that could make it very isolated since you can simplify specify the name of your connection argument and the number of arguments you expect, like a normal command.

This also would allow people to try different authentication mechanisms.

Additional "processing instruction" type

In a project I've been hacking on, a redis-like frontend for LMDB, I implemented and extended the Redis protocol with a few additional types.

One of these I think may interest you I'm calling a "processing instruction". It allows the client to send additional metadata along with a single command. Currently I am only using this functionality to allow the client to specify a different database for a single command (LMDB, like Redis, supports multiple databases within a single environment). So rather than having the client do:

• SELECT 0
• do some operation on db 0
• SELECT 15
• do some operation on db 15
• SELECT 0 -- reset the client state.

I can just explicitly SELECT 0 once, and when I need to run a single command against DB 15, I send (along with the command) a processing instruction that indicates the DB for that command should be db 15:

• do some operation on db 0
• (include processing instruction to use db 15) + do some operation on db 15
• do another operation on db 0

Another usage for processing commands could be to set/modify the expiration time on the key being operated on.

Set replies - unique?

Looking at the set replies, can you have non-unique items?

Example - is this valid?

~3<CR><LF>
+redis<CR><LF>
+redis<CR><LF>
+redis<CR><LF>

Doubles: reasons for disallowing exponential form

The specification indicates, for doubles:

Exponential format is invalid.

However, this means that very large and very small values are now very verbose on the wire; additionally, many frameworks make not using E format challenging. This is also contrary to previous redis handling, which allows E formats, for example in the INCRBYFLOAT documentation there exists the sample:

INCRBYFLOAT mykey 2.0e2

For example, IEEE754's max/min values would be over 300 bytes, to transmit a theoretically 8 byte payload.

I'd like to challenge whether this is a good idea, preferring to allow exponential form when appropriate. It is also notable that some platforms make it actively hard to not use E form.

Idea: backwards compatible way to drop the CR

What if the initial HELLO message could either end with CRLF or just a single LF, and which one is used determines wether the rest of the connection will use CRLF or LF as a line break. This way clients could continue using CRLF if that is simpler, or switch to just using LF with RESP3 if desired. The biggest downside is that it makes the server code slightly more complicated, and such functionality would probably only be used by RESP3-only clients.

I'm not saying this should necessarily be done, but if removing the unnecessary carriage return is desirable, this could be a way to do it.

All in all the saving of one byte did not made enough sense in light of a more complex client implementation

I'm curious if there is any data on what the actual overhead of the carriage return is on real-world connections. If the bulk of messages are sufficiently small, one byte could make a difference.

[NOTICE] Redis RESP3 specification moved

Please note that the Redis project, which implements a large subset of the RESP3 spec, will be maintaining its independent copy of the 74adea5 as committed by the Creatore in this repository.

The Redis RESP3 spec can be found at https://github.com/redis/redis-specifications/blob/master/protocol/RESP3.md

🙇 k10xby

Nested streamed aggregated data types

The spec does not specify if the newly added streamed aggregated data types may be nested or not. It is of course possible to implement this in a parser but it complicates the logic for "event based languages" such as JavaScript since it's required to keep lots of state around to keep the parser fast.
Limiting the streamed aggregated data type to only be allowed in the top level would simplify that.

Monitor mode works with other data as well

@antirez in the spec you state the following:

The connection can only be used for Pub/Sub or MONITOR once setup in this way

However, when being in MONITOR mode Redis will still respond with regular commands if send by the client being in MONITOR mode. The data is send in addition to the actual monitoring information. I actually have a implementation where I use a regular expression to distinguish regular data from the monitoring information (there is no other way AFAIK).

This should likely either be officially supported or the support for that should be removed. I was not sure if it's best to open here or directly for Redis. Please just move the issue if you feel it is best tracked elsewhere.

Add a distinct type for keys?

Hi,

According to http://antirez.com/news/125 the goal with RESP3 is to be able to have relatively dumb clients which offer a very simple interface such as result = redis.call(“GET”,keyname); and don't need to know anything about the command it sends.

As the current maintainer of redis-rb, I'm totally behind this as it would indeed make my life much easier, and I'm really eager to be able to delete tons of code.

However I question wether such simplicity will really be achievable in practice. With a regular Redis setup it will indeed be possible, but as soon as you use Redis in a distributed manner, say with Redis cluster, the client do need to know which elements of the command are keys, so that it can hash them and select the proper target.

So ultimately I'm afraid the client will still have to maintain a list of commands, as well as their possible arguments as to be able to extract keys.

Hence why I wonder if keys could be a distinct type from strings, with their own prefix such as # or @.

This way, rather than redis.call("GET", "foo"), the interface would be something like redis.call("GET", redis.key("foo")), allowing the client to extract the keys without having to know anything about the command signature.

Guaranteeing reply vs pubsub might be too constraining

however the order of the commands and their replies is not affected: if a command is called, the next reply received will be the one relative of this command

I think this is how Redis will work due to its architecture, but I think it is unnecessarily constraining. I think the spec could avoid this guarantee and the clients should support the pubsub before the command reply.

I'm also not sure if the guarantee can actually be made, since the client may not know when the command was received and a push event may have been triggered by another connection writing concurrently.

Blob Error Code & Verbatim Type code question

The spec says (for simple errors)

The first word in the error is in upper case and describes the error code. The remaining string is the error message itself. The ERR error code is the generic one. The error code is useful for clients to distinguish among different error conditions without having to do pattern matching in the error message, that may change.

and (for blob errors)

The general form is !<length>\r\n<bytes>\r\n. It is exactly like the String type. However like the Simple error type, the first uppercase word represents the error code.

So am I correct in assuming that the error code must

1. be upper-case only, or not contain anything other than A-Z0-9-_ or some other limited character set?
2. be at the start of the error string?
3. be separated from the rest of the error by a space ?

---

For the verbatim type, does the 3 character type marker have constraints? I assume it can't be <CR> or <LF>, but perhaps keeping it within the ASCII printable range?

Carification of attributes: arity, key uniquity, and permitted key types

Arity

For any (following) element, it seems clear that attributes can be omitted or present, but it isn't
explicit whether they can be repeated. For example, we know this is valid:

|1<CR><LF>
    +ttl<CR><LF>
    :3600<CR><LF>
:3<CR><LF>

but is this valid

|1<CR><LF>
    +ttl<CR><LF>
    :3600<CR><LF>
|1<CR><LF>
    +priority<CR><LF>
    :7<CR><LF>
:3<CR><LF>

and if so (my personal preference would be "no, that isn't valid"), is it semantically identical to:

|2<CR><LF>
    +ttl<CR><LF>
    :3600<CR><LF>
    +priority<CR><LF>
    :7<CR><LF>
:3<CR><LF>

or does it need to be possible to tell between the two scenarios?

---

Uniquity

How is the following to be interpreted?

|2<CR><LF>
    +ttl<CR><LF>
    :3600<CR><LF>
    +ttl<CR><LF>
    :9001<CR><LF>
:3<CR><LF>

is this:

• illegal
• valid, last value wins (so: ttl = 9001)
• valid, all values should be preserved (so: ttl = [3600, 9001] in some way)
• something else?

---

Permitted Key Types

(a lot of this may cross over into map more generally)

In a lot of platforms, it may be hard to reliably implement maps that can take arbitrary types as keys, as the equality semantics are very awkward. In reality, I can see a lot of clients would want to restrict handling to string-like types, as that is what most attributes are going to be keyed as. Now, sure, clients could choose to just treat all keys as blobs, but that has other problems.

As a pragmatic note, it may be sensible to consider whether it is actually required for attributes (and maps?) to allow keys that are, say, integers, doubles, big integers, in the case of attributes, it might even be prudent to limit it to simple strings, i.e. +foo

Verbatim string format

I find that the Verbatim string has a fixed-size header of 4 bytes with the type and a colon is inconsistent with the rest of the spec. I also think it affect readability that the header has no visual separation to the first word of the string. I think it would be better if it uses the <CR><LF> separator.

For example

=16<CR><LF>
txt<CR><LF>
Some string<CR><LF>

Maps: are keys reliably unique?

The specification indicates that they should be treated as a dictionary / hash structure, but it isn't clear what should happen if duplicate keys are encountered. The specific scenarios are shared with #35

and could be re-stated:

How is the following to be interpreted?

%2<CR><LF>
    +ttl<CR><LF>
    :3600<CR><LF>
    +ttl<CR><LF>
    :9001<CR><LF>
:3<CR><LF>

is this:

• illegal
• valid, last value wins (so: ttl = 9001)
• valid, all values should be preserved (so: ttl = [3600, 9001] in some way)
• something else?