Base64-encoded binary for responses is very inefficient and waste both CPU and memory resources. There should probably be a better way to transfer data between GatewayD and plugins without converting it back and forth several times.
This is a multi-repo project that involves changes to GatewayD, SDK and plugins at the same time.
A select query on a table with 158k rows returns 64.59 MB of binary data to the client. The same exact binary data is base64-encoded into 304.81 MB of stored cache value, a 4.7x increase in memory usage and a lot of time lost in processing.
Progress Report
No. 1
I've made some progress in using a message that can contain JSON as well as binary message as bytes array. The JSON encoding and decoding is heavily tangled and intertwined into the code and it affects almost all the repositories, including, but not limited to, the following:
The above changes are still inconsistent and unstable and needs a lot of work to make it actually work.
No. 2
I made some more progress. Now the gatewayd process and the plugins don't crash, but messaging between the core and the plugins don't work as expected. I am trying to fix them as they appear. The code is ugly for now, but works. The moment I get feature-parity with the old message, and the responses and other binary objects are transferred in the Message.Blob[x].Value, I'll refactor and fix the ugly parts and release a new version.
No. 3
The request/response messages are correctly sent to the plugin and back to the core. The missing part is Redis. Redis cannot handle byte array, so the data has to be converted to an string representation, which again converts the data to base64-encoded string, which is not ideal. I also tried the gocache.Marshaler to use msgpack, but it also adds another conversion layer. I have to either convert the bytes to string, which increases time complexity, or find another way to deal with this issue. Using gocache.Marshaler with msgpack seems to be the way to go. So far, the separation of JSON from BLOB proved to be very useful, both usability- and performance-wise.
Another idea I tried to quickly find a way for this performance bottleneck is to compress/decompress the data before saving/restoring it from cache. I used Snappy, which is quite fast compared to alternatives. The result is really impressive, compared to the current storage requirements of the base64-encoded string. Compressing the base64-encoded response from the 158k-rows table resulted in 114.06 MB of data being stored in cache, which is 1.76x increase in size. However, the transferred data between the core and the plugins stay at 4.7x (or 304.81 MB).
No. 4 ✅
The obvious: protocolbuffers/protobuf#3078.
This led to me copying the struct.proto and extend it with a BytesValue. So, the resulting output is an extended struct that contains an extra field type: BytesValue. This removes a lot of copying and creating custom message with multiple fields to handle bytes, like I did in the above changes in the use-bytes-array-for-efficiency branches.
I tried the above, but there are still issues:
This works as expected, except it hangs on large blobs. That is, the transferred from DB to client and its cached value are almost equal in size (almost, as in 99%, probably due to CRC, headers or memory overhead added by Redis). On the other hand, the code is more simplified and one doesn't need to deal with multiple different variations introduced by solution no. 1. If I can fix the large blob transfer issue, this is the way to go forward and it has maximum compatibility with google.protobuf.Struct.
No. 5
The Envoy Proxy approach: https://blog.envoyproxy.io/dynamic-extensibility-and-protocol-buffers-dcd0bf0b8801.
They used a TypedStruct, which contains an arbitrary JSON serialized protocol buffer message with a URL that describes the type of the serialized message. This is very similar to google.protobuf.Any, instead of having protocol buffer binary, this employs google.protobuf.Struct as value.
This is not what I want, as it is the same thing as google.protobuf.Struct, and wouldn't support byte array.
No. 6
So far I tried running GatewayD on Debian 11 (on WSL2) and it caused a lot of challenges on Microsoft and custom kernels I compiled. I ran it again with the main (stable) branch without the changes on Debian 12 (testing) and it behaved differently. The difference in behavior was 1) faster load time, especially with cached data and 2) no intermittent unresponsiveness.
Verdict
This should be picked up again at a later stage.
Resources
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent