Byte Abstraction

[rsocket]

Byte Abstraction

[benjchristensen]

cc @NiteshKant as this is similar to what we have been discussing for ReactiveIPC.

[benjchristensen]

@tmontgomery and I were discussing the approach used in Agrona that Aeron uses.

[tmontgomery]

Will have to see if Agrona UnsafeBuffer can wrap a Netty ByteBuf. That would be the easiest if possible.

[NiteshKant]

If the intended buffer API in reactive socket is an interface, then I think i can make netty work in a way that instead of netty's ByteBuf we can have a ByteBuf impl that additionally implement our interface and we don't need to do an additional object allocation for wrapping per ByteBuf use.

[tmontgomery]

I think there are two pretty viable options. Agrona has a read-only and a read-write interface for accessing by index into a buffer. The interfaces are:

• https://github.com/real-logic/Agrona/blob/master/src/main/java/uk/co/real_logic/agrona/DirectBuffer.java
• https://github.com/real-logic/Agrona/blob/master/src/main/java/uk/co/real_logic/agrona/MutableDirectBuffer.java

Implementing a concrete class for MutableDirectBuffer to netty ByteBuf should be straight forward.

Alternatively, it might be easy to add a new wrap function for UnsafeBuffer that can take a ByteBuf. This is simpler and also provides all the concurrent operations as well.

Either of these is pretty straight forward to do and gives appropriate data layout access.

[NiteshKant]

I see this Buffer abstraction we will have, show up in a few places:

• Inside the currently existing Message abstraction to get the frame data.
• Inside the client/server protocol classes for initiating different request types in some form.
• For any generic encoder/decoder we write. eg: we will write an encoder/decoder for the payload data which will help to be generic, irrespective of which underneath network library we use.

Thus, I see the usage to be internal and external and so I would say we do not use an external library (agrona in this case) to represent that abstraction. We can pick the methods that makes sense from agrona and netty and define a Buffer abstraction which can eventually be defined in ReactiveIPC.

Once we have this interface, it is certainly possible to have an implementation for netty which will eliminate copying or wrapping of the underlying ByteBuf per use.

[tmontgomery]

Whether it be Agrona or not, it will need to support a few things for completeness (learned from developing Agrona buffers):

• means to wrap with no allocation:
  • byte[] (part of interface)
  • java.nio.ByteBuffer (including direct) (part of interface)
  • netty ByteBuf (left to concrete impl)
  • Optional: off-heap array (left to concrete impl)
• ByteOrdering optionality with nativeOrder default. (part of interface)
• Java primitives (int, long, float, double) at an arbitrary index. (part of interface)
• buffer put/get at an arbitrary index to/from any form supported by wrap above. (part of interface)
• String encode/decode at an arbitrary index with length ByteOrder. (part of interface)
• Optional: access via Unsafe and ability to disable per access bounds checks.

[benjchristensen]

We chose ByteBuffer for better-or-worse.