Custom starting offset

[elodina]

Custom starting offset

[joestein]

We could probably start storing an offset for every minute or something so you can re-wind at any minute in the stream. We are working on some refactoring in that part of the code over the coming weeks. I think so maybe we can try to hook something up that would work best.

[sajal]

For my use case, I could manage the offset myself if there were a way to specify which starting offset to use when starting up (or re-balancing).

A logic like x offsets ago is also fine. Say when starting up or (rebalancing) the newest offset is x... it could process from offset x - n where n is something configured.

[baconalot]

+1 a common usecase I have is priming a job with some historic data, but not the complete available dataset.

[baconalot]

Only now do I see the problems here. With a single consumer pid it would work fine if we had something in config like ForceSetParitionStart = 1234.
Flow would then be:
-pid 333 (consumer.go) start
-set offsets to 1234
-start consuming

But... with multiple pids (chronos anyone):
-pid 333 (consumer.go) start
-(333)set offsets to 1234
-(333)start consuming
-pid 334 (consumer.go) start
-(334)set offsets to 1234
-(334)start consuming
-(333) -> reprocess messages

Also how would this be configured. Cant be single int, since there can be any topic/partitions in the conf's group.

For now I am just going to use a helper that can set a commit manually for me:

    zkconfig := go_kafka_client.NewZookeeperConfig()
    zkcoord := go_kafka_client.NewZookeeperCoordinator(zkconfig)

    err := zkcoord.Connect()
    if err != nil {
        fmt.Errorf(err.Error())
    }

    tp := go_kafka_client.TopicAndPartition{}
    tp.Topic = "some_topic_name"
    tp.Partition = 213
    err = zkcoord.CommitOffset("some_group_id", &tp, 123)
    if err != nil {
        fmt.Errorf(err.Error())
    }

[sajal]

Isint Offset tied to a partition and not at topic level? or do you mean to process single partition in multiple pids?

As long as each partition gets processed by single pid(my usecase), is there any issues in using x - n as starting offset?