icgc-argo/workflow-raccoon

hacktoberfest

workflow-raccoon's Issues

Workflow Raccoon implementation

Cleaning up stale / orphaned pods/jobs

When collab crashes, we are left with inconsistent states. Jobs left hanging
When we resume, we rip through these and manually emit a “failed” event into relay to kill these.

Scaffolding has been done here: icgc-argo/workflow-roadmap#189
Design: https://wiki.oicr.on.ca/display/icgcargotech/Workflow+Garbage+Collection

Implementation Details:

• find stuck runs in rdpc gateway and update their state
• find and clean stale pods in kubernetes
• find and clean stale config maps in kubernetes

collection events

• The garbage collection should emit a "SYSTEM_ERROR" event for the workflow run when it needs to be cleaned up to a correct state.
• RUNNING/CANCELLING/INITIALIZING need to be scanned

Feature Request - Clean up rdpc tasks

Use run date filters to ignore new runs

• Use date filters to filter out runs before a certain number of days
• make the age of pods to consider configurable
• this will allow raccoon to run while we have active runs