current sendfile fetch is racy

[erleans]

current sendfile fetch is racy

[tsloughter]

Are you positive that the file size would get bumped intermediately during the writing processes flush to disk?

We may also want to look at using some other limiting factor, like a max send config, as well.

[evanmcc]

My thought was something like this:

1. logfile stands at 10 messages, all in the same file.
2. fetch request for offset 4 arrives
3. write request for message 11 arrives, it's a large message
4. first chunk of 11 is flushed to files
5. fetch request finds the current file size
6. second chunk of 11 is flushed
7. sendfile sends 4-10 and first chunk of 11

Is there something that makes this interleaving impossible?

We should definitely allow max bytes. I'd like to add offset ranges for fetch, but it doesn't seem to be supported by the protocol.

[tsloughter]

That is my question, would a flush of message 11 result in this interleaving, or would the size and file not actually be updated until the whole chunk was written.

[evanmcc]

It's probably a per-filesystem thing. This suggests that anything over 4k is potentially a problem: http://stackoverflow.com/questions/29866047/whats-the-atomic-disk-write-for-a-linux-filesystem

[tsloughter]

Ok. Kafka also has a limit on the max message size.

[evanmcc]

urgh, complicated config: http://stackoverflow.com/questions/21020347/kafka-sending-a-15mb-message

anyway, that's way bigger than 4k, and I don't imagine a 4k limit on message size is going to make people happy. also for speed, we're likely going to want to concatenate messages to write them to disk, which could end up breaking them anywhere.

[tsloughter]

I wouldn't be surprised if kafka just sends the extra data and the client is expected to drop it.

The only reason to use the size instead of the lookup is speed.

[evanmcc]

yeah, dropping it is a valid option, but inefficient for large messages, since you can potentially send and resend the data several times.

[tsloughter]

Yea, it is what kafka does. Not that we have to do the same here (we wouldn't even have to consider if we want to break protocol since it would be the same still) but I don't know that doing the lookup is worth it.

[evanmcc]

A thought: if we're passing the latest ack'd txid back up the chain (to allow read load to spread out), we could also pass the offset limit back up and keep it handy.

[tsloughter]

We'd have to keep it handy in the gen_server (bottleneck) or an ets table (meh).

[evanmcc]

Store it in the state for the currently written file, which we have to fetch anyway.

[tsloughter]

The index file? We don't read any other file besides with sendfile and that goes straight to the client.

[evanmcc]

there was some offline discussion; closing as not really important right now, as the client can always limit its over-read by specifying smaller byte limits, and the read optimization that I mentioned isn't likely to be needed for our usecase.