Coalesce user events

[hashicorp]

Coalesce user events

[mitchellh]

We talked about this in IRC today. We came up with a few points:

• You probably never want to replay events (even unique ones) on a manual serf join. It may be useful to add a flag to Serf.Join to ignore events from the push/pull sync.
• Coalesce, like you said here, by default. Add a flag if you don't want it to coalesce. serf event -repeatable foo

[jpfuentes2]

After our conversation last night since it's likely in the vast majority of use-cases you do not want the replay events, do you think it'd make more sense to make serf join ignore these events by default and the flag would instead accept them?

[mitchellh]

@jpfuentes2 Yeah, we plan on implementing this very soon. The coalescing itself is a harder problem. Armon described it to me last night:

1. You have nodes A+B in one Serf cluster.
2. You have nodes C+D in another Serf cluster.
3. Both clusters live on their own for some period of time, using events and what-not.
4. You join B to C. Now the cluster becomes A+B+C+D.
5. It is easy to coalesce the events on C, but then they all still get broadcast out to D (by A or B), which is much harder.

[darkone23]

looking forward to serf join --ignore-history or something similar

+1

[vmalloc]

Just out of curiosity, how many events are backlogged? For sure it can't be infinite, right? Couldn't find any hint in the docs about that

[mitchellh]

@vmalloc I believe 512, currently. This is hardcoded and non-configurable.

[mitchellh]

@vmalloc Actually that might be 1024, its either 512 or 1024.

[armon]

@vmalloc I guess it depends on what you mean by backlogged. If you sit in a tight loop firing serf events, then the transmit queues will basically become infinitely backed up. They start to generate warnings about queue depth, but will otherwise not stop you. In terms of anti-entropy and ensuring message delivery, we have a config called EventBuffer (https://github.com/hashicorp/serf/blob/master/serf/config.go) which defaults to 512. EventBuffer controls how many events which we have received we keep around to perform anti-entropy. This is the buffer that gets 'replayed' when a partition heals or node joins, etc. Hope that makes sense!

[armon]

As of 331142d the ground work for this is in. When Join is called, ignoreOld can be used to avoid processing older messages. Events can be coalesced using configuration setting when starting serf, and it can be enabled/disabled on a per-message basis when an event is fired.

What is left now is to integrate this up into the agent and CLI. For now, the behavior of the agent/CLI is the same for compatibility. @mitchellh do you want to take this?

[mitchellh]

Assigned to myself, I should have a PR for this very shortly.

[mitchellh]

@armon Question: do you think user events should be coalescable by default? i.e. serf event foo == foo is coalesced? I think in the common case, coalescing is okay and may be expected behavior. But wanted to check.

[armon]

@mitchellh I think we coalesce by default yes, it seems like the expected case