kuhnuri/kuhnuri-queue

Kuhnuri queue specification and reference implementation

kuhnuri-queue's Issues

Add filtering to jobs end-point

Add parameters to filter the output for the jobs end-point. Filters should include:

• state

Support source and result ZIP storage

Support uploading source ZIP and downloading result ZIP directly from queue.

Persist job logs instead of passing to queue

Instead of passing the log to the queue on submit, the worker log should persisted just like output resource. Logs can be very large at debug level, thus they cannot be returned as part of the submission JSON.

At task level, this could be

{
  "output": "s3://bucket/output.zip",
  "log": "s3://bucket/output.log",
}

Add support for sequence of transtypes

Currently job can only contain a single transtype that should be used to process the job. This should be extend to be a sequence of transtypes that need to be ryn in order. For example in pseudo code

{
  transtype: [
    { name: "graphics" },
    { name: "pdf2" },
  ]
}

Queue should first run graphics transtype and then pdf2. When both transtypes have completed, the job is considered done.

This enables use cases such as

• handle graphics conversion and DITA processing in separate workers
• convert DITA to FO in first worker and render FO to PDF in the second worker
• convert DITA to CHM source files on a Linux worker and convert to CHM on a Windows worker.

By separating different type of processing of separate workers, the workers can be run on different machines per CPU or IO usage. Similarly a part of processing that requires licensed software or a given platform can be run in a single common worker while doing generic DITA processing in multiple workers.

Add password support with BCrypt

Add support for password on worker registration. Hash passwords with jBcrypt

Timestamp all state transitions

Add timestamps for all state transitions, i.e. for starting processing and finishing processing.

Add support for worker specified data

Workers should be able to store key-value pairs of data when job is submitted back.

Check worker status after timeout

A job is given to a Worker. When timeout x has occurred, ping Worker to see if it's still alive.

If Worker doesn't respond, change job state back to queue.

Expose all job information in REST API

Expose all job information in REST API.

Add pagination to jobs end-point

Add pagination for the jobs end-point via page parameter and the Link header.

References

• GitHub pagination

Priority for queue items

Add priority field for queue items. Field could be signed byte, with convention to use zero for default priority.

Persist done and errored jobs to archive

In memory-backed queue persist done and errored jobs to JSON archive file and keep only queue and process jobs in memory.