An application developer must be able to control the 'exit codes' returned by the container [BATCH-40]

[spring-projects]

An application developer must be able to control the 'exit codes' returned by the container [BATCH-40]

[spring-projects-issues]

Dave Syer commented

One way (the most natural) to do this would be to change the interface of Module/Task to return an exit code. However, in most cases developers do not implement this interface directly, so it gets messy.

One important question - what proportion of batch jobs need this feature? I'm guessing it's low - 5% maybe.

[spring-projects-issues]

Dave Syer commented

It has been bothering me lately that a failed job leaves no record in the database otehr than a BatchStatus value. It would be better to put an exit code and also a message (e.g. stack trace in the case of a failure) in the *Execution. That would imply a signature for RepeatCallback and Module/Task that returned a value object like ExitCode, containing a true/false for un/finished, interger for code, String(s) for messages and/o details.

[spring-projects-issues]

Lucas Ward commented

I absolutely agree, and think it makes total sense for Repeat Callback to return a value object. Although, if there is a boolean and messages ExitCode may not be the best name, how about ExitStatus?

Also, how would a developer control what was returned? Especially if they created an item provider that just gets plugged in?

[spring-projects-issues]

Dave Syer commented

public class MyItemProvider extends AbstractItemProvider implements StepScopeContextAware {
public Object next() {
...
context.setAttribute("MY_STUPID_EXIT_CODE", new Integer(112));
}
}

public class MyRepeatInterceptor extends RepeatInterceptorSupport {
public RepeatStatus after(RepeatContext context) {
StepScopeContext stepContext = StepSynchronizationManager.getContext();
if (stepContext .hasAttribute("MY_STUPID_EXIT_CODE")) {
return new ExitStatus(true, context.getAttribute("MY_STUPID_EXIT_CODE").intValue());
}
return ExitStatus.CONTINUABLE;
}
}

[Edited because APIs changed, e.g. RepeatContextAware was phased out in favour of core domain centric version StepScopeContextAware.]

[spring-projects-issues]

Deep Kapadia commented

Couple of comments:

I disagree with calling it ExitStatus. Staus is something that indicates a current state of affairs of a given step or a module. An ExitCode is more generic since all it is, is a return value to the caller of the batch job containing information about the result of the job. I am assuming most of these jobs will be called using a scheduler such as cron or be wrapped within a shell script or a batch script. In my opinion all "well written" shell scripts return exit codes to indicate success/failure. There may be multiple scenarios for success or failure.

I am not sure if 5% or less batch jobs will require this feature. The number should be higher in my opinion. In fact, I would recommend that every batch job return an exit code that provides information about how it completed as a good practice.

Also, I agree with Dave's idea of including a method on the interface that allows a developer to return an exit code. While all may not implement the interface, the idea is to have a provision or a facility built into the interface.

[spring-projects-issues]

Lucas Ward commented

Deep,

The reason ExitStatus was proposed is because the ValueObject also indicates whether or not the callback is finished or not, so it is in effect a status, and part of that status is the Exit code, and potentially a stack trace. While I agree that it's not like the Status saved for each job, using status as a suffix in similar scenarios has a lot of precendence within spring (I.e. TransactionStatus, etc)

I'm interested in hearing more about your comment: "I am not sure if 5% or less batch jobs will require this feature. The number should be higher in my opinion. In fact, I would recommend that every batch job return an exit code that provides information about how it completed as a good practice. "

I would be interested in seeing some business scenarios for differing codes on batch jobs. I may be missing something (and please point out if I am) but it seems like ExitCodes are the only way of communicating between a batch job and the scheduler, so their real use would be to control flow. Otherwise, it's just reporting and storing the ExitCode in the status tables will suffice (something we plan on doing anyway) The only scenario's I can see are:

• All records processed successfully
• Job completed successfully with skipped records(which would indicated to the scheduler that another 'cleanup' job needs to be run)
• Application specific control flow (i.e. because of the data processed, job C should be run next instead of Job B)

The first two can be easily done at the architecture level, the last one is tough, because it is what requires developers to tell the architecture what to return. It seems like the need for a developer to do that wouldn't be very often, which I think is where Dave's 5% is coming from.

[spring-projects-issues]

Deep Kapadia commented

The scenarios are pretty much what you describe Lucas.

I guess I did not understand what Dave meant when he made his comment. I understood (or misunderstood) from his comment that he was talking about a need for a batch job to have exit codes. Thank you for clarifying. Yes the need for the third scenario is sparse.

I missed it the first time around that you were referring to the ExitStatus as a value object which contain the exit code. Now that you have clarified it, it does make more sense.

[spring-projects-issues]

Wayne Lund commented

Paulo, Lucas and I have been discussing this topic at some length and wanted to weigh in on the following:

1. ReturnCode - We agree that the return code needs to be a Value Object but we feel it needs to be returned from the module. This should work even when called from the RepeatCallback since it already returns a RepeatStatus per your latest update. ExitStatus may be a better name than RepeatStatus.

2. We don't really believe we need the logic for the ItemProvider as demosntrated above because it doesn't matter because our interface is with the module. In the future if we need to provide this we can add but for now we have what we need from the return value coming from the module.execute() method.

3. Lucas raised the issue about ensuring that the Module remains stateless. ExitStatus should ensure this but we also had the question about whether we care if its stateless because its being created on the stack in the execution within the thread of the StepExecutor. We may need to discuss if we're misunderstanding how modules will be created within the scope of a step.

[spring-projects-issues]

Dave Syer commented

1. ExitStatus/ReturnCode absolutely should be returned from Module/Tasklet/Thing. I think Lucas plans to do that as part of BATCH-24.

2. You don't need the ItemProvider/RepeatINterceptor pattern if you implement Module. I would rather you did not need to implement Module, but if that's the way you want to add an exit code it's still an option.

3. Module is not created by the StepExecutor so we can't confine it to the stack. This was a major issue until we hit on the idea of scope="step" to regulate the lifecycle of Module (or anything in step scope).

[spring-projects-issues]

Tommy C. Trang commented

Background: A static batch schedule where all the jobs to be run in the night are pre-determined. The batch schedule changed daily but jobs are not dynamically scheduled during the batch run.

A few solid examples for scenario #3 that Lucas mentioned where we used an exit code in the past:

1. Inbound File Not Received - Here we have a separate FTP job to move the inbound file from the FTP server in the DMZ into our Document Management Server (DMS) and a separate job to read the file from the DMS to process the file. If the file is not present by an agreeable time, the FTP job returns a unique exit code to cancel the reader job. The reader job failed if the file is not present and we don't want an error in this scenario because receiving the file is optional for the night.

2. Inbound Empty File / Outbound Empty File - Similar to scenario #1 but we have the option of how to handle the file based on the business requirement whether to process the empty file or to send an empty file to a third party partner. The FTP job to send / receive the file and the job to generate / process the file are two separate jobs.

3. Outbound Central Print - Here we prep, bundle, and send any document for central printing to our printing facility. If there is no document, then we send a unique exit code to cancel the bundling job and the FTP job.

4. Notification - In certain pre-determined scenarios, we send a unique exit code for the scheduler to take specific actions such as paging someone with information.

5. Unique errors to track separately - There are a couple of unique errors we needed to track separately with a different exit code than the generic exit code of "1" for error and "0" for success.

[spring-projects-issues]

Lucas Ward commented

I created the ExitCodeExceptionClassifier, which can be passed to the SimpleStepExecutor to map exceptions to exit codes, which can then be stored before rethrowing the exception. The only remaining piece is Batch-24, where an ExitCode to integer translator will need to be created.