Eventual consistancy

[adlnet]

Eventual consistancy

[fugu13]

I'd lean more towards letting LRSs give an approximate eventual consistency window in /about -- after all, the main situation it's of interest is in syncing scenarios. It can only be an estimate, or a large over-stating to be absolutely sure, after all.

[bscSCORM]

@fugu13 I'm against anything that would force us in to overly-large estimates, because I hate to build waste into a standard, and as it's a completely reasonable thing to sync frequently (we're only getting new records after all), any window beyond what's actually used will be wasteful.

OTOH, I can't think of any reason why putting the window in the /about would require a larger estimate than applying it when returning statements and calculating a header -- so ignoring your comment about over-stating, info probably makes sense.

[fugu13]

Over-stating's going to be pretty necessary (there's really no way around it if we want the time to reflect a near-certitude, since in almost all cases the time will be far better than the 95th/99th percentile), but it doesn't preclude staying up to date and using frequent requests, it just indicates the minimal overlap in times for those requests needed to ensure nothing is missed.

[bscSCORM]

@fugu13 it was the "large" over-stating that concerned me. I hope systems will state more like the 99.999th percentile, but I also hope that's still seconds not minutes.

[fugu13]

Hmm, at 5 nines? maybe around ten seconds for most sane systems. Though many systems may be capable of handling extremely high load at a large delay (for instance, dump ten million statements in a minute into a system that's been configured to handle ten thousand statements a minute at 10 seconds or less at 5 nines and you'll see that delay skyrocket).

[bscSCORM]

That could be a reason for including this value in a response header
when fetching statements, the system could adjust how large the window
needs to be due to load.

On Fri, Mar 29, 2013 at 4:42 PM, fugu13 [email protected] wrote:

Hmm, at 5 nines? maybe around ten seconds for most sane systems. Though
many systems may be capable of handling extremely high load at a large
delay (for instance, dump ten million statements in a minute into a system
that's been configured to handle ten thousand statements a minute at 10
seconds or less at 5 nines and you'll see that delay skyrocket).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/120#issuecomment-15659706
.

[fugu13]

That's a reasonable point, though it's going to be not so straightforward to estimate in many cases (the causes of long delays tend to be idiosyncratic). Lets keep poking at this into next week sometime, at least, I don't want to rush to a conclusion (and this'll slot in easily, it's a very esoteric if useful bit).

[nhruska]

TODO: add to spec - suggest activity providers set 'Timestamp' so they can sort in a timeline view as opposed to allowing LRS to set it
-QUESTION: require (MUST or SHOULD?) LRSs to keep up to date with time server?
-NOTE: timestamp is meant to be the time the stmt was received (if set by the LRS - SHOULD be set by client if its important for sequencing)
-QUESTION: do we need a 'Received' time - when the stmt is first received by any LRS in the cluster?

NOTE: 'Stored' can change when stmts are synced between LRSs

[fugu13]

I think the first NOTE is a little unclear. Timestamp is meant to be the time the event recorded by the statement occurred, and if unspecified, is set to the time the statement is received as a 'best approximation' of that. Activity providers SHOULD set timestamp, I think. I don't think we need to have a 'received' time, and the simplest reason against it is that a cluster of LRSs isn't a well defined concept.

[creighton]

I agree. I think that timestamp and stored are sufficient. And that if a client cares about the sequence of statements, it should set the timestamp value.

As for the window of storage latency, I defer to those that understand the issue better.

And finally about where or how to inform a client of the latency, I think either approach (/about or header) is fine with me. I do recall that the ETag process was described in the spec in the case that the client did not have access to the header info. With that in mind, perhaps a non-header based approach is best. However if we want to keep it accessible on a per request basis, could we add it to the StatementResult object? So it would have fields of 'statements', 'more', and 'storagelatency' or whatever?

[bscSCORM]

@creighton I like the StatementResult idea, though I'd have the LRS do the calculation, so I'd call it something like: "statementsAvailableThrough" which would indicate with "reasonable certainty" that all statements which would wind up with a stored time <= the "statementsAvailableThrough" time had been considered in this query.

@fugu13 I'm pushing again for this approach since it seems like it allows the most implementation flexibility to the LRS, while also providing the information where it is most needed to clients. LRSs that wish to use a "fixed window" as you suggest could simply subtract that fixed window from their system time when responding to queries. LRSs that want to adjust the window based on load/etc can do so too.

[bscSCORM]

@fugu13 one more idea: say the LRS SHOULD NOT return statements with a stored time later than the latest stored time of any statement which might not yet be available in query results.

IOW, instead of returning "statementsAvailableThrough" as part of query results, simply internally apply that value as an "until" filter if no "until" filter is specified.

[bscSCORM]

After discussing with @fugu13 , added #171

@creighton : note that we don't have the same worry here about clients not being able to read the header since this property is really only relevant to clients that are trying to maintain a replicated copy of an LRSs statements -- which is something less capable clients are unlikely to need to do.