ducharmemp/ironworker

ironworker's Issues

Create Data<T> wrapper for things that should come from the state

Right now we just imply that everything that comes after the first argument comes from the state. What I'd like to see is an API that looks like this:

fn my_job(db_conn: Data<DbConn>, payload: Message<u32>)....

So basically we can specify args in any order and we can also impl a trait (let's call it FromJob) for any type that needs to either grab things from the state or transform the payload in some way.

Implement way of enqueueing jobs within jobs without OnceCell

I don't want users to have to maintain a global for enqueueing jobs, let's allow them to extract the Application instead (or something like an Application)

Improve the Queue UI for Axum

The queue UI is pretty barebones right now. We should add/expose additional metadata to the UI and allow for things like retrying failed jobs

Integrate UI and Setup with Actix

Setting up a project should be straightforward, let's make it simple with an integration into actix.

Rework broker to return standard error types from core

Right now we use unwrap everywhere, let's get rid of that. We should return standard errors everywhere we can.

Todo: how to chain an error from say Redis into a standard error with thiserror?

Tombstone Cleanup

When a job dies (typically if a worker gets killed), the job that it was processing never gets re-enqueued. There should be some way of having a worker take some time to clean up after its dead comrades.

How/should it be synchronized? This smells like a scheduled task, and scheduling isn't something that's been figured out yet.

Add postgres queueing mechanism

I'd like ironworker to be fairly plug and play for databases, postgres seems like a good option to start because of its ability to lock (and I'm more familiar with it than other DBs). We should create a postgres broker and a way to run migrations via the broker to set up the job tables.

Scheduled Jobs

Related (and I think foundational to) #5

I'd like to have the concept of scheduling a job at some point in the future. I don't think this would work with a rpush style mechanism in redis, so there's probably some research that needs to be done in order to figure out the best approach.

If we have a scheduled job, only one worker can pick up the job, making maintenance tasks much easier (and opening up the possibility of cronjobs/recurring jobs in the future)