hrdag/mse-prox

code that manages processing MSE scripts through Amazon SQS

mse-prox's Issues

SQS timeout bug

When a stratum takes longer than the SQS timeout period, SQS moves the stratum message from "in flight" back to "queued." That means that the stratum is assigned to another thread while the first thread may still be running.

A worker thread cannot tell the difference between a stratum that has been abandoned (perhaps because it was dequeued but then its worker failed) and one that is currently being computed but taking a long time.

The problem is that these very long-lived strata end up being computed more than once. In fact, any stratum that takes RUNTIME > SQS_TIMEOUT will be reinserted in the queue and reassigned ceiling(RUNTIME/SQS_TIMEOUT) times. These are the hardest-to-compute strata, so this can very greatly extend the total runtime.

This needs attention. Maybe a central watcher script can harvest the currently-running strata ids, and somehow broadcast it to workers so they avoid dequeueing a currently-running stratum?

create watcher script

Need a script that runs on cev (or whatever the machine is where the estimates will be pulled out of the queue and stored). This should:

• observe all AWS instances, tracking n_procs and cpu utilization
• shut down or SIGHUP estimate.sh/estimate.R processes that are calc'ing a sha1 already in /datos