batch size

[cdenneen]

batch size

[ph]

This is a needed improvement, having multiple input using the same sincedb isn't an option for now. Since this part of the code isn't threadsafe. One way to workaround it is to use multiple inputs with different prefix and of course different sincedb_path.

[cdenneen]

@ph different prefix? Can you explain?

[DanielRedOak]

See #25

[cdenneen]

@ph @DanielRedOak I'm running latest logstash 5 and have 7 s3 inputs with different prefixes to parse each month of this year individually but the inputs are taking forever to load 30 days worth of data.
Obviously it's a lot of data but there needs to be something to speed this up.
Whether that's having more input threads than cores so they can batch up quicker, much much smaller interval so it cycles back faster, something... my 4 core box for the most part is at least 80-90% idle so it's definitely not processing to it's fully potential.
If you need any info for config, api stats, etc I'd be happy to share and willing to test any config changes in order to improve performance considerably here.