summerbatch/summerbatch

Summer Batch is a lightweight, reliable, efficient, open-source batch framework for the C# community.

Summer Batch

Summer Batch is a lightweight, reliable, efficient, open-source (Summer Batch is distributed under the Apache 2.0 license ) batch framework for the C# community.

Its design has been driven by the concepts exposed in the JSR-352 specification (which is a java colored but exposes universal batch concepts).

Purpose of Summer Batch:

• Build new batch solutions to be run on Microsoft®-based environments
• Migrate smoothly their batch legacy from mainframe to modern Microsoft®-based environments

Some of its key features are:

• Repeatable and customizable batch jobs
• Multi step jobs, with simple step sequences or conditional logic between them
• In-memory or persisted job repository
• Support for a Read-Process-Write logic, as well as arbitrary batchlet for a more complete control on the behavior
• Chunk processing with checkpoint management and restartability
• Step partitioning used for parallel processing
• Database readers and writers, with support for Microsoft® SQL Server, IMB® DB2 and Oracle® databases
• Flat file readers and writers
• Easy mapping between the readers and writers and your domain classes
• Batch contexts at step level and job level
• XML design for the main batch architecture, C# design for the properties

Get the source and build Summer Batch

You need to use Visual Studio, version >= 2013 to build Summer Batch from the sources.

Clone the git repository (using whatever protocol you prefer).

Simply build the downloaded solution using VS.

Get the binaries

The simplest way to get your hands over Summer Batch binaries is to use the NuGet package repository.

Getting started

To get started with your first Summer Batch project, please follow the getting started guide.

This will lead you, in a few minutes, to get your first Summer Batch project up and working.

You can browse the samples repository to get your hands over sample Summer Batch projects.

Going further

To dig deeper into Summer Batch, we recommend to browse the following sites :

• Summer Batch main site
• Reference Guide
• Summer Batch API Doc

Want to Contribute ?

You can use the github Issues appliance to ask for improvements or to file bugs you might encounter. To take any other contact (pull requests, questions, etc), please write to the provided email address.

Or

1. Fork it
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request

summerbatch's Issues

DatabaseBatchItemWriter<T> failed to insert null values into DB

I've got an error when DatabaseBatchItemWriter insert a record with null values. Please see details below.

Item class:

public class Item
   {
       public string Id { get; set; }
...
       public DateTime? CompleteDateTime { get; set; }
   }

Register:

// Writer - Step1/DatabaseWriter
            container.StepScopeRegistration<IItemWriter<Item>, DatabaseBatchItemWriter<Item>>("Step1/DatabaseWriter")
                .Property("ConnectionString").Instance(writerConnectionstring)
                .Property("Query").Value(Queries.DataWriterSql)
                .Property("DbParameterSourceProvider").Reference<PropertyParameterSourceProvider<Item>>()
                .Register();

//Error message:
2018-10-09 12:51:32.0381|DEBUG|Summer.Batch.Core.Step.Tasklet.TaskletStep|Applying contribution: [StepContribution: read=2, written=0, filtered=0, readSkips=0, writeSkips=0, processSkips=0, exitStatus=EXECUTING]
2018-10-09 12:51:32.0381|DEBUG|Summer.Batch.Core.Step.Tasklet.TaskletStep|Rollback for Exception: SqlException : The parameterized query '(@id nvarchar(32) expects the parameter '@completedatetime', which was not supplied.
2018-10-09 12:51:32.0381|ERROR|Summer.Batch.Core.Step.Tasklet.TaskletStep|Transaction will be rollbacked because of an unexpected exception.
2018-10-09 12:51:32.0851|ERROR|Summer.Batch.Common.Transaction.TransactionScopeManager|Transaction has been rollbacked.

SqlException when multiple jobs start at once

Hi !
In our environment we need to run two jobs in parallel.
I do get a SQL Exception.
System.Data.SqlClient.SqlException: 'Transaction (Process ID 55) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.'

I think it's also related to
https://jira.spring.io/browse/BATCH-2147

SummerBatch to work with Unity 4.01

If our application components are referencing Unity 4.01, it causes an issue for SummerBatch integration which is looking for version 3.5 for things like IUnityContainer and SingletonRegistration.

TransactionScopeManager: resources registered while a transaction is already active are not enlisted

The transaction scope manager allows registering resources to automatically enlist them when a transaction is opened, but if a transaction is already active the resource is not enlisted in it.

This is an issue with transactional writers that will wait a commit confirmation that they will possibly never receive.

SortTasklet - Unstable Sort vs Stable Sort

We are currently modernizing a legacy JCL/COBOL application and using SummerBatch. Through our batch testing we have discovered that the SummerBatch sorting functionality does not match the legacy functionality. While the outputted file does sort correctly by the supplied SortCard attribute information, we were still seeing differences. It appeared that the legacy sorting was retaining the original file sort orders better.

After further investigation into the SummerBatch source code, we found that the code is doing what is called an “unstable sort”, where the legacy code was doing a “stable sort”. The specific code in question can be found in class Summer.Batch.Extra.Sort.Sorter, in both methods SortAndSave() and InMemorySort().

The following snippet of code is performing the unstable sort in those methods:

// Sort the records
records.Sort(Comparer);

The List.Sort Method (IComparer) is being used. If you follow the link to the MSDN documentation you will find the following in the “Remarks” section:
“This implementation performs an unstable sort; that is, if two elements are equal, their order might not be preserved. In contrast, a stable sort preserves the order of elements that are equal.”

Using this information we ran some tests in which we replaced the Sort() method with LINQ’s OrderBy() method, which performs a “stable sort”. These tests produced the expected same sort order as the legacy sorting.

The following snippet of code was used to replace the example snippet above:

var listStableOrdered = records.OrderBy(x => x, Comparer).ToList();
records.Clear();
records.AddRange(listStableOrdered);

The Enumerable.OrderBy<TSource, TKey> Method (IEnumerable, Func<TSource, TKey>, IComparer) is being used in this case. If you follow this MSDN link, you will find the following in the “Remarks” section:
“This method performs a stable sort; that is, if the keys of two elements are equal, the order of the elements is preserved. In contrast, an unstable sort does not preserve the order of elements that have the same key.”

If an unstable sort was the intention of the SortTasklet, I think the functionality should be expanded to indicate if a stable sort should be performed instead.

Thanks,
Todd

Default ConnectionStringSettings registration missing in the sample

We enabled job repo persistence support in SummerBatch. However, we needed to supply a ConnectionStringSettings instance to the container like this in MyFirstBatchUnityLoader.cs inside your sample application. Otherwise , DbJobRepositoryFactory can not be resolved due to missing connection string parameter. Something is probably missing from LoadConfiguration in UnityLoader base class. Can you validate the issue?

    public override void LoadArtifacts(IUnityContainer container)
    {
        //Connection string
        var writerConnectionstring = ConfigurationManager.ConnectionStrings["Default"];

        container.RegisterInstance<ConnectionStringSettings>("Default", writerConnectionstring)
        ...
    }

Any plans for .NET Core support?

We like this framework but are moving towards .NET Core. Is there any plan to port (might necessitate a Unity replacement with something like Autofac since I don't think Unity supports .NET Core yet either).

FileUtilsTasklet throws an error when copying a single file

When copying a single file using FileUtilsTasklet an Exception is thrown stating that the file does not exist even it actually exists.

How to write parent child relationship in database?

Can we write parent child in database? if yes, can you give me sample code.

IProcessReader<...> in process does not get resolved

I try to use a

[Dependency("CustomerAdapter")]
public IProcessReader CustomerAdapter { get; set; }

when I try to access it remains "null"

public class BatchUnityLoader: ContextManagerUnityLoader
{
private const string SelectSalesOrderHeaderQuery = @"SELECT [SalesOrderID], [CustomerID] FROM [Sales].[SalesOrderHeader]";
private const string SelectPersonQuery = @"SELECT p.FirstName, p.LastName FROM Sales.Customer c
INNER JOIN Person.Person p
ON c.PersonID = p.BusinessEntityID";
public override void LoadArtifacts(IUnityContainer container)
{
base.LoadArtifacts(container);

        var writerConnectionstring = ConfigurationManager.ConnectionStrings["Default"];

        var readerConnectionString = ConfigurationManager.ConnectionStrings["Default"];

        container.RegisterInstance<ConnectionStringSettings>("Default", readerConnectionString);
        //SalesOrder Reader
        container.StepScopeRegistration<IItemReader<SalesOrder>, DataReaderItemReader<SalesOrder>>("CreateInvoice/SalesOrderReader")
            .Property("ConnectionString").Instance(readerConnectionString)
            .Property("Query").Value(SelectSalesOrderHeaderQuery)
            .Property("RowMapper").Instance(SalesOrderRowMapper.RowMapper)
            .Register();

        //CustomerReader
        container.StepScopeRegistration<IItemReader<Customer>, DataReaderItemReader<Customer>>("CustomerReader")
            //.Property("Resource").Resource("#{CustomerID[0]}")
            .Property("ConnectionString").Instance(readerConnectionString)
            .Property("Query").Value(SelectPersonQuery)
            .Property("RowMapper").Instance(CustomerRowMapper.RowMapper)
            .Register();

        //Invoiceprocessor
        container.StepScopeRegistration<IItemProcessor<SalesOrder, Invoice>, InvoiceProcessor>("CreateInvoice/Processor")
            .Property("JobContextManager").Reference<IContextManager>(BatchConstants.JobContextManagerName)
            .Property("StepContextManager").Reference<IContextManager>(BatchConstants.StepContextManagerName)
            .Register();

        //ProcessReaderAdapter
        container.StepScopeRegistration<IProcessReader<Customer>, ProcessReaderAdapter<Customer>>("CustomerAdapter")
            .Property("StepContextManager").Reference<IContextManager>(BatchConstants.StepContextManagerName)
            .Property("Adaptee").Reference<IItemReader<Customer>>("CustomerReader")
            .Register();


        container.StepScopeRegistration<IItemWriter<Invoice>,
           DatabaseBatchItemWriter<Invoice>>("CreateInvoice/DatabaseWriter")
           .Property("ConnectionString").Instance(writerConnectionstring)
           .Property("Query").
               Value("INSERT INTO dbo.Sales.Invoice(NAME,Address,City, Total)"
               + " VALUES (:Name,:Address,:City, :Total)")
           .Property("DbParameterSourceProvider")
           .Reference<PropertyParameterSourceProvider<Invoice>>()
           .Register();
    }

.Net Core release

Can you provide a release for .net core ?

Unsafe Object Deserialization

SummerBatch contains code which deserializes object via instantiating a BinaryFormatter. It may be vulnerable to other attack categories, such as information disclosure or remote code execution.

Disposable instances created by the step scope are not always disposed at the end of the step

When a dependency is registered in the step scope an instance is created per step execution. At the end of the step execution instances implementing the IDisposable interface should be properly disposed.