Add Pagination Support

[ebaizel]

Add Pagination Support

[thehenrytsai]

The only intended use of the API currently is for a Sidtree node (sidetree-core) to fetch ALL the transactions starting at a certain point in time.

The intention is to let the blockchain implementation return all transactions that are known at the time of asking immediately so Sidetree can start processing them, there is no point IMO for the sidetree-core to specify the number of transactions to return in because: 1. Would the blockchain layer need to wait if the transaction count known falls short of that number until the asked size is met? 2. If it returns the exact number of transactions as asked, does the Sidetree node need to ask for more immediately? 3. The Sidetree node really just wants to know ALL transactions available, it might as well specify a very large number, like 1000000, is seems rather arbitrary whatever that number is. Thus I opted for just having a flag in the response indicating if there are more transactions or not, and let the blockchain layer decide how many to return, with expectation that the layer doesn't return 1 at a time if it knows of 1M. In the case of 1M transaction hashes, it can choose to return all of them in one shot, or 100K at a time, sidetree-core probably don't care all that much.

Hopefully it makes sense. Happy to discuss further.

[ebaizel]

Thanks Henry that makes sense.

I'm thinking in addition to the ethereum smart contract, we can track the transactions and ipfs hashes in a server cache (memory or localdb) and make lookups much quicker. Or do you think we need the guarantee that we're reading this directly from the blockchain?

[thehenrytsai]

Hi @ebaizel, cache is a major component in the sidetree-core architecture, we are currently calling it the DID Cache. You can see from the architecture diagram that lookups/resolves are intended to hit the DID Cache for perf reasons as you suggested. The description of the DID Cache can be found further down the same implementation doc, but more details can probably be provided. In fact, I plan to merge the first in-memory implementation of it into master sometime today, with the intention that persistency probably using localdb (coincidentally) to come within a week or two. It is by no means the final version, but you can take a look at the pull request here and give comments if you want.