Comments (8)
Peter Zozom commented
I have created new flat file output sources (see attached patch).
- SingleFlatFileOutputSource has same functionality as FlatFileOutputSource, but it is refactored to be extendable
- SplittableFlatFileOutputSource extends SingleFlatFileOutputSource and allows to specify 'max record count'. When record limit is reached new file is created. Output Source attaches part number to filename as extension so file names are filename.txt.001, filename.txt.002. Output source also supports rollback to last commit point located in different file than currently opened file (e.g. we are processing filename.txt.005 and last commit point was somwere in filename.txt.002)
from spring-batch.
Lucas Ward commented
Peter, I'll take a look at the solution as soon as possible. At first glance it seems to me that writing a simple wrapper for the input source would be the simplest approach. The wrapper would delegate read calls to it's inputSource and call close when the limit is reached, and open again on a new resource. However, extending may be an easier to configure solution. In my mind this is somewhat related to BATCH-16. The approach taken in the xml input sources was to use a static utils class to help with restarting, etc. However, it seems to me that common file handling concerns for reading in from both xml and flat files could be handled using a common abstract class. This would allow a wrapper, such as one that could be created to fix this issue, to work for both xml, flat file, and anything else that extended the common abstract file input source interface.
from spring-batch.
Lucas Ward commented
I understand your point about extending the output source so that you can span rollbacks and commits across files. However, I'm not sure if we should be doing that. Personally, I would prefer that any limit given simply means that we will close the file at the first commit point after the limit is reached. However, I would be interested to hear if anyone has a use case that requires an exact file size or record count.
from spring-batch.
Lucas Ward commented
Moving to 2.0, as there seems to be no requests for the feature.
from spring-batch.
Robert Kasanicky commented
Is this still a feature we care about implementing? It's being pushed forwards for almost a year now...
from spring-batch.
Dave Syer commented
We scheduled it for 2.0.0.M3. If it isn't really difficult I think we should do it. But I would concentrate on more important things first (this could always go into 2.1).
from spring-batch.
Robert Kasanicky commented
There is now basic implementation commited.
TODO:
-strategise naming of created output resources
-test with StaxEventItemWriter
from spring-batch.
Robert Kasanicky commented
There is now MultiResourceItemWriter that works with both FlatFileItemWriter and StaxEventItemWriter and has customizable suffix creation. One can set an item count limit per resource and a new resource is created after this limit is exceeded, but only on chunk boundary.
from spring-batch.
Related Issues (20)
- Schema Migration with Flyway HOT 2
- DELETE CASCADE on Foreign Keys HOT 6
- Spring Batch step write_count less than read_count and filter and skip counts are all zero HOT 1
- Incorrect Chunk property value in implementation of ItemWriter write method HOT 3
- 5.1.2 Backported issues HOT 1
- 5.0.6 Backported issues HOT 1
- remote partitioning doesn't work if you're using graalvm
- Access Job Description
- Kotlin data class support for `FlatFileItemReaderBuilder` HOT 4
- Deserialization of JobParameters throws exception
- Default value for ignoreWarnings in JdbcCursorItemReaderBuilder does not align with documentation HOT 1
- JdbcPagingItemReader - When using sortKeys with alias, I think it should paging by column name rather than alias in the select clause.
- MockStatic and MockConstruction not working in spring batch test cases HOT 1
- ListItemWriter<T>::getWrittenItems has bad wildcard return type HOT 1
- Data class support in JdbcCursorItemReaderBuilder & JdbcPagingItemReaderBuilder HOT 2
- Discrepancy in Thread Safety Documentation for AbstractPagingItemReader Subclasses
- Improve Error Messages in JobParametersBuilder methods
- The table field type in the SQL Server database is ntext and synchronized to the Oracle database, resulting in invalid column types in the corresponding NCLOB
- Add static variables to prevent typos in the metadata tables string path HOT 1
- Deprecate the usage of LobHandler HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spring-batch.