Add time limits to process_item? (Possibly just in MapBuilder?)

[materialsproject]

Add time limits to process_item? (Possibly just in MapBuilder?)

[shyamd]

I think I need to fundamentally rework the parallelization to use asyncio. There are a few places in the runner that are just not very efficient or flexible.

[mkhorton]

Isn't asyncio more for IO-bound tasks specifically? Not sure if our builders are IO-bound as much as CPU-bound

[shyamd]

asyncio was designed for that but basically, the builder infrastructure is the asynchronous execution of 3 things: get_items, process_item, and update_targets that are manually synchronized at key points. The futures framework used in python plugs right into asyncio, but you get added thigs like cancelling, wait shielding, etc.

In terms of future scaling, this could also be a useful framework to think about scaling to AWS. If get_items yielded a coroutine to grab data, you could theoretically get infinite scaling with something like lambda + ddb

[shyamd]

Added in #54