[QUERY] Can excessive memory usage by reactor.util.concurrent.SpscArrayQueue while consuming from an EventHub be controlled? Or is this a bug?

[ChrisCollinsIBM]

[QUERY] Can excessive memory usage by reactor.util.concurrent.SpscArrayQueue while consuming from an EventHub be controlled? Or is this a bug?

[anuchandy]

Hi @ChrisCollinsIBM, we optimized the memory allocation of Processor in version 5.18.0 azure-sdk-for-java/sdk/eventhubs/azure-messaging-eventhubs/CHANGELOG.md at main · Azure/azure-sdk-for-java (github.com), this should lower the memory usage that you are seeing.

Each partition has a dedicated connection (link) to the service, the in-memory queues exist for each partition-receive. Each partition-receive is managed by an instance of partition-pump-x-x, so as you observed, allocation adds up based on the number of partitions-receive hosted in one machine.

[ChrisCollinsIBM]

Thanks for the prompt response @anuchandy, I did find #38572 when I went digging into the EventHub Messaging history so I presume that's the fix you're referring to.

Since we're using the prefetch defaults (300) what would you suggest for a reasonable batch size? I see 100 tossed around in many discussions using a 3:1 prefetch:batch ratio but maybe that was cache related. But then in some other places I see 10 as a batch. Some guidance on this would be great, thanks!

[anuchandy]

Hello @ChrisCollinsIBM, sorry for the late response. @conniey and I discussed this. We don’t have a one-size-fit-all recommendation for tuning prefetch and batch size for optimal memory, there is also a third variable of expected event(s) size. Our suggestion is to run the application (with actual event processing logic) and tune these values to achieve expected throughput. While doing this exercise, identify an appropriate value to set for max heap size (-Xmx). The idea is, once the application run reaches a steady state with expected throughput, force a full GC using tools such as JConsole, check how much memory is occupied after the full GC. You want to size the heap such that only ~30% is occupied after full GC; use this value to set the max heap size (-Xmx). Size the host (e.g., container) memory to have an "additional ~1 GB" of memory for the "non-heap" need for the JVM instance.

[anuchandy]

Closing this, refer previous comment.