Bulk lookup proofs

[facebook]

Bulk lookup proofs

[slawlor]

Hey @eozturk1, so in the MySQL storage layer we already have a caching scheme implemented. By using the preload_nodes_for_insertion you're effectively using batch get operations rather than individual get ops. Anytime a record is retrieved, we're appending it to the cache so you don't need to worry about manually inserting them into any cache, it's auto-magic ;)

The bulk-get is MUCH fewer db ops which is why it's helpful to retrieve them all before, then after that it's just memory operations which are super cheap. Even though we're double-getting data, it's still way more performant and makes the code transparent to the AKD dev.

[eozturk1]

Looking at the preload_nodes_for_insertion function, it retrieves the nodes from storage but I am not sure how it achieves the caching we would like to have. I might be missing something here but the function does not seem to keep the nodes, e.g., in memory. How do we want to cache the nodes for bulk lookup proofs?

[eozturk1]

Ah, that's perfect! Just saw the TimedCache.:) I'll use the same batch ops for bulk proofs.

[eozturk1]

I'll close this one since the bulk of the task has been completed in #163. The remaining optimizations are tracked in #166, #167 and #169.