KTable State Persistence

[nodefluent]

KTable State Persistence

[krystianity]

Hi @coderroggie for now I would suggest to fall back to node-sinek and use NConsumer (with manual batch mode) and NProducer for distributed high volume situations.

I simply dont have the time and focus to build persistency based on a solid KTable implementation for this lib at the moment - as we have a lot of new challenges that I have to tackle with updates to other libs first.

[krystianity]

Hi there @dm261395 and sorry for my late reply.

Indeed the current KTables implementation in this lib has the exact flaws you pointed out.

• KTables focusses purely on a key/value relationship of an inner kafka message's payload value
  and not on the actual key to value of kafka messages like the Java client (however it is possible to pass the whole message back as value to the kv function that you pass as argument when creating a KTable instance)
• The backup through Kafka topics to re-store KTables has just been implemented as foundation so far and has not been fully implemented

Both points have historical reasons.. as we mainly needed a fast and stable streaming solution when I built this lib a while ago. Feel free to open any PRs regarding KTables or Windows.

Regards,
Chris

[JRGranell]

Hi @krystianity,

Would the backup through topic you describe in your second point be possible with just changes to this lib? Would there anything from the underlying libs (such as 'librdkafka') that would need to support it?

Thanks

[krystianity]

Hi @JRGranell the backup is just based on producing to a topic and re-consuming back from that in case of re-starts etc. there is no need for changes on the low-level clients for that. It requires a bit of deeper Kafka knowledge, as it can be a bit tricky to get single instances in a shared deployment to consume entire topics to rebuild the KTable state in memory or in the DB; however personally I see this as definitely possible to accomplish in the lib.

[coderroggie]

@krystianity how many 🍻 would it take for you to implement the persistence in this lib? Persistence would allow for us to scale up our microservices when using this lib... without it it seems rough to use in high volume situations.

😄