Rebalance can leave some workers with no partitions

[faust-streaming]

Rebalance can leave some workers with no partitions

[jkgenser]

@bobh66 : Just wondering, why did you revert your changes to fix the partition assignment problems raised in this issue?

[bobh66]

Hi @jkgenser - I needed to do more testing with the partition assignment changes and I didn't want to hold up releasing the other two fixes which are also important. The existing partitioner logic will intentionally leave a consumer without any active partitions, which is also validated by existing tests, so in order to "fix" that I need to change the way the tests work. I'm reluctant to make too big a change in this area given how much of an impact it can have. I'm going to restrict the partitioner changes to only force a "balanced" configuration when there are no tables in use and the table_standby_replicas option is set to 0. That will limit the initial impact of the change so we can see how it works, and then decide whether to expand the implementation or not. Since the partitioner will default to promoting a standby partition to active, it is still possible to have empty assignments even with my changes, so it's better to leave that code alone for now. Hopefully this makes sense.

[taybin]

Has this issue been resolved?

[xaralis]

Has this issue been resolved?

@taybin We're currently experiencing this so I guess this isn't resolved yet.

[bobh66]

At this point there will always be scenarios where this can happen - see this comment for more details on exactly what was fixed - #97 (comment)

Basically you need to "enable" the fix by setting table_standby_replicas to 0.