Does this library handle backpressure?

[pmiddend]

Does this library handle backpressure?

[davidmoten]

You are quite right. By using interval source we are not supporting backpressure. I'll remove that bit from the README. We are essentially polling the file system for changes. If we did implement backpressure then the obvious strategy is to poll only when there are requests but we then open up our window for missing file events. Are you overwhelmed by file change events or are you just improving my docs?

[pmiddend]

Currently I'm merely being careful, actually. :)
A good first step would be to not use interval and instead create a Flowable that calls onNext when the FileWatcher signals changes (so use blocking take instead of a regular poll). I think this is quite the "reactive" way of doing things, because then, you don't need prescribe a particular scheduler or do multithreading in your library. The user can decide if blocking should be done in a particular scheduler if needs be (in my scenario, I'll happily use the current thread), and changes get pushed as soon as they arrive. There's no backpressure handling needed in your library, the user can decide.
Thoughts?

[davidmoten]

Yeah that's fair enough. I can have a look at providing an alternative to interval so that users can choose between take and poll approaches.

In terms of pros and cons of take vs poll

• take is a blocking call (blocking a thread is something we try to avoid in reactive programming generally) and if we do block a thread it should not be on the computation scheduler
• the better latency of take may help with not losing events if a lot of file events happen in a very short time because the events are dequeued faster. Note that take or poll may lose events from the internal buffer of the OS if the processing of events happens on the same thread and is slow compared to the rate of production of events. On Linux I believe you can configure the buffer size (perhaps through a JVM property).

In terms of backpressure with the take method it probably makes sense to return an Observable which the user can convert to a Flowable with whatever buffering/dropping strategy they want. In fact a drop strategy is present already in the WatchService in that the OS has an internal queue of limited (but configurable) size.

I'll have a look at how the API might look and run it by you.

[pmiddend]

Good idea about returning an Observable. Are you sure you want to support both take and poll though? Doesn't seem like poll has lots of advantages...

[davidmoten]

Are you sure you want to support both take and poll though? Doesn't seem like poll has lots of advantages...

Yep. poll approach is my preference for my use cases because of the context switching overhead of that blocked non-computation thread unblocking frequently (assuming lots of activity).

[davidmoten]

This conversation has been useful. I suspect the API needs to return Observable everywhere because backpressure on the source is not practical. That way the user can convert to Flowable as they wish and apply their backpressure strategy (buffer, drop, etc). I'll still attempt to support poll and take.

[pmiddend]

Glad to have helped. I understand your use case for using poll to limit "wake-ups". What I'm doing is simply tailing a file. For this, using latest as the backpressure method should be sufficient, right? My reasoning is that the tailing logic only needs to know when to check for updates to the file. Not sure about log rotation, though.

[davidmoten]

latest might work, the library currently collapses MODIFY and OVERFLOW events based on time-based sampling (TODO make sure users can specific the sample time from builder paths) and all CREATE and DELETE events come through untouched (might help in log rotation). I don't have a log rotation unit test but would be nice to have (PR welcome).

I've made a lot of API changes in #2 to accomodate both non-blocking and blocking techniques and some other improvements as well. Check PR #2 for details. Your review is welcome if you have time. I'll merge in a couple of days and release.