gamaops/hfx-bus

Redis backed high frequency exchange bus

hfx-bus's Issues

v2 GA

• Add unit tests
• Add more examples
• Add architecture overview
• Rewrite benchmarks

Examples from README.md are broken with v1.1.2

With v1.1.1, the example works. Only v1.1.2 has problems.

Steps to reproduce the behavior:

1. start consumer
2. start committer => nothing happens.
3. stop both
4. start consumer => lots of exceptions:

Bus S4KaCspAW1-3f-KU started!
{ event: 'healthz:ping:pending',
  asyncListener: [AsyncFunction],
  error:
   { Error: Undefined payload key "ping"
       at Object.message.load (/home/adi/leaflet/cc-live-map/node_modules/hfxbus/lib/hfx-bus.js:318:12)
       at process._tickCallback (internal/process/next_tick.js:68:7) data: { key: 'ping', message: [Object] } },
  args:
   [ { messageId: '_kpGUaDD0oiG_OKk',
       trackerId: 'yVBM5NZLjPPb27ZQ',
       streamId: '1557251758314-0',
       replyTo: [Array],
       groupName: 'healthz',
       streamName: 'ping',
       isAcknowledged: false,
       isResolved: null,
       _release: [Function: _release],
       ack: [AsyncFunction: ack],
       load: [AsyncFunction],
       save: [AsyncFunction],
       drop: [AsyncFunction],
       purge: [AsyncFunction],
       resolve: [AsyncFunction],
       reject: [AsyncFunction],
       ping: null } ] }
{ event: 'healthz:ping:pending',
  asyncListener: [AsyncFunction],
  error:
   { Error: Undefined payload key "ping"
       at Object.message.load (/home/adi/leaflet/cc-live-map/node_modules/hfxbus/lib/hfx-bus.js:318:12)
       at process._tickCallback (internal/process/next_tick.js:68:7) data: { key: 'ping', message: [Object] } },
  args:
   [ { messageId: '8AQzDFL3atkk4R7U',
       trackerId: 'ZuGDFH7V3RxN_vcb',
       streamId: '1557251763305-0',
       replyTo: [Array],
       groupName: 'healthz',
       streamName: 'ping',
       isAcknowledged: false,
       isResolved: null,
       _release: [Function: _release],
       ack: [AsyncFunction: ack],
       load: [AsyncFunction],
       save: [AsyncFunction],
       drop: [AsyncFunction],
       purge: [AsyncFunction],
       resolve: [AsyncFunction],
       reject: [AsyncFunction],
       ping: null } ] }
[...]

Please complete the following information:

• OS and Version: Ubuntu 16.04
• NodeJS Version: 10.15.3
• Redis Version: 5.0.4
• Redis Driver and Version: ioredis 4.9.3

Client side partitioning

Client side partitioning

I have the following use case:

1. I need to have Redis with HA setup and persistence
2. The ideal way to scale the environment is horizontally
3. Redis Cluster has some drawbacks:
   1. You can't use pipeline or any multi keys commands (without prefixing)
   2. A reliable way to scale Redis Cluster horizontally is difficult to achieve when we're dealing with persistence

As we can see here Redis tells us that clients can be used to shard data across nodes, so I think the ideal feature achieve this setup is:

1. Have an infrastructure with multiple Sentinel+Redis (masters and slaves) units, in that way one "HA unit" doesn't need to be aware of each other
2. HFXBus should implement the possibility to create an instance that is able to route commands to clients based on hashing algorithm: crc16(routingValue) mod numberOfHAUnits should give us the index of client that the command should use
3. This implementation will route clients using consistent sharding so pipelining is possible due the static routing of commands through the same client

I'll start the work on this feature and it'll be a minor change to keep our current major v2.

Create stress tests

HFXBus need stress tests, we need to plan which scenarios will be tested. The test must differentiate times by HFBus processing time and Redis/Network response time.

Using the same connection to ack jobs and read from groups

When the same connection is used for both, the blocking call xreadgroup will block any other commands.

FYI - stream implementation used in Redis-Comander

Hello,

just want to let you now that your implementation for stream support was used to add stream support to Redis-Commander, a redis web ui (joeferner/redis-commander#351). The PR of the other user is not merged right now but will be soon.
(current WIP for integration: https://github.com/sseide/redis-commander/tree/streams)

I'll add some more license/copyright informations to the file at our project and will add a link back to your project from our readme. Hope its ok to use it (your license implies it). But just wanna let you know in case there are some objections or helpful hints...

Best regards,
Stefan Seide

Improve unit tests quality

• Some tests are only calling functions to see if they don't throw an error.
• Need to unit test Claimer

Could you please explain, a bit, the design of hfx-bus?

Hi,

Could you please explain the general flow in terms of Redis structures and key names?

I see a Redis Stream ("ping"), okay..., looking through the code I also see some pieces related to PubSub but PUBSUB CHANNELS * doesn't list anything.
A short architecture paragraph would be great.
...something like https://www.slideshare.net/RedisLabs/redisconf18-building-lightweight-microservices-using-redis starting from slide 57

Thank you

PS: also 'hfxbus' is said to be the default namespace, shouldn't the stream name be hfxbus:ping ? or is this related to NodeJs/EventEmitter/RedisConsumerGroups/PubSub... what's the purpose of heartz?

I'm just learning Redis & NodeJS & Javascript & Co... so apologies for the some of the, maybe, dumb questions

Question: Possibility for batchprocessing? (XREADGROUP COUNT > 1)

Hi,

thank you so much for your hard work!
There aren't a lot of projects utilizing redis streams at all and I love how very complete your implementation is!
I just read through the documentation and didn't find a way to fetch multiple items at once.
I know it's very event driven, but is this still possible somehow?

It would be awesome if up to X items could be read at once (non blocking) in intervals, for preprocessing them before processing and acknowledging them as a batch.

(Use case is some minor serialization during processing inside consumers running in parallel)

Thank you very much!

Distributed routing

To scale Redis horizontally we have to support distributed routing:

• Producers should send messages using the job's ID as the routing key
• Consumers must read from all nodes using a round-robin algorithm to rotate the number of messages to be acquired from each node

Handle message.load inexistent key error

Describe the bug
message.load method is not handling unexistent payoad key properly.

To Reproduce
Try to load an unexistent payload key.

Expected behavior
Thrown a custom error class telling that this key doesn't exist, allow an option to just ignore the key.