checkpoint_limit behavior for kafka vs kafka_franz inputs

[joshgarrett-hellofresh]

checkpoint_limit behavior for kafka vs kafka_franz inputs

[Jeffail]

Hey @joshgarrett-hellofresh, there's a difference in the implementation where the kafka_franz input uses the checkpoint limit to disable new records on the next poll for partitions that reach the limit. However, this doesn't explicitly prevent processing in parallel in cases where an individual poll obtains records beyond the limit. This means the lower the checkpoint limit is set for the kafka_franz input the less accurately it'll be honored.

One of the requirements for marking it as stable rather than beta is to have the checkpointing behaviour moved up the batching mechanism within the plugin implementation and therefore match the documentation exactly. However, this isn't a huge priority as there are other ways of guaranteeing processing ordering of partitions via config even though they're more complicated.