ConcurrentModificationException error

[rfoltyns]

ConcurrentModificationException error

[rfoltyns]

Couple of things worth mentioning:

• AsyncBatchDelivery has default batchSize set to 1000 and deliveryInterval set to 1000; at 100k/sec that you mentioned in previous issue, I'd not recommend sending 100 bulks every second as the ES ingestion process may start to suffer; see performance-related articles referenced here; also JestHttp connection pools and timeouts may need some adjustments
• io.searchbox.Bulk.Builder uses LinkedList under the hood and there's already an issue created for this in Jest project; however, org.appenders.log4j2.elasticsearch.jest.BufferedBulk.Builder uses ConcurrentLinkedQueue instead; you can switch to JestBufferedHttp until I have a fix for the ConcurrentModificationException; switch may lead to partial log loss when conditions for mentioned exceptions occur, instead of losing the whole batch
• consider using AsyncLogger; it offers much higher logging throughput (regardless of appender used underneath); see Log4j2 docs for more info
• eventual switch to JestBufferedHttp mentioned before, would require BufferedSource as input; currently, it can be created only by JacksonJsonLayout:PooledItemSourceFactory, which would make your KeyValuePair decoration impossible (I'll probably add support for additionalProperties in future releases)

[rfoltyns]

This bug was a mix of io.searchbox.core.Bulk being backed by LinkedList and issues in BulkEmitter batch delivery - lack of AsyncLogger exposed them instantly.

@execretor Could you retest with the latest release (1.3.2)? Also, could you try the AsyncLogger? It will vastly improve the logging throughput and latency.

[rfoltyns]

@execretor This issue was fixed in 1.3.2. Did you manage to retest it yet?

[hellcats88]

Sorry but our code Is now in uat testing environment and for now we are not planning a new release before some months.

If we will find the error another time we will try the new fix

[rfoltyns]

Closing due to lack of activity.