Clean up --cleanup

[coyotemarin]

Clean up --cleanup

[coyotemarin]

I think the best way to handle this is have cleanup refer to what we do when the job succeeds, and have a separate option, cleanup_on_failure for when the job fails.

[pbharrin]

would these options be directories/files to remove or more complex actions?

[coyotemarin]

Yeah, cleanup just removes files. Here's what I was imagining:

The default behavior would be the same, on success (cleanup), cleanup everything, and on failure (cleanup_on_failure), clean up nothing.

cleanup and cleanup_on_failure would both take lists of strings describing what to clean up, with these possible values:

• ALL
• NONE
• SCRATCH
• LOCAL_SCRATCH
• REMOTE_SCRATCH
• LOGS

On the command line, you could specify something like --cleanup-on-failure=LOCAL_SCRATCH,LOGS or --cleanup=NONE

Does this seem like a good design? Is it something you'd be interested in working on? :)

[pbharrin]

Yeah I would be interested in working on that.
is the default behavior NONE, for both --cleanup and --cleanup-on-failure?

I had in mind a more functional approach where you could pass in function pointers that would do whatever you liked in the event of a success or failure. Perhaps you would like to be notified via text message if a job failed. My approach would not work when executing jobs from the command line, so we can go with your approach.

[coyotemarin]

Cool, it's yours (though github 500s when I try to assign it to you, darn!)

The default behavior is ALL for cleanup and NONE for cleanup_on_failure (which is the same as the current behavior, which we call IF_SUCCESSFUL).

The main point of cleanup is just to make sure temporary files don't pile up over time, so a simple approach ought to work. For example, if you want to be notified of a job's success or failure, just wrap it in something that looks for a nonzero return code or an exception (depending on how you run it).

Having said that, can you think of other use cases where more flexible cleanup actions would be helpful?

[coyotemarin]

Oh, and please work from the development branch; it's different enough that merging in changes made against master might be tricky.

[coyotemarin]

Ah, had to add you as a team member. Congrats, you are the official owner of this Issue. :)

[irskep]

It would also be nice if the cleanup functions could be called on their own w.r.t. a job ID from the command line. Typing s3cmd del --recursive can get tedious when debugging. python mrjob/tools/emr/cleanup.py --behavior=ALL j-MY_JOB_ID would be much better.