Container exited with a non-zero exit code 137 | Out of memory

[Vivek-Merugu]

Container exited with a non-zero exit code 137 | Out of memory

[Vivek-Merugu]

The problem happened because there wasn't enough memory allocated to the Spark executor. After increasing the Spark executor memory, the issue was resolved.

Hence, closing this issue.

[cerveada]

What makes you believe that this issue is caused by Abris?

[Vivek-Merugu]

Thank you for your response. I appreciate your assistance in addressing this issue. Here's a breakdown of the situation:

Old Build Overview:

• The initial Spark Streaming Scala code processed JSON messages from Kafka Topic A.
• Operations were performed on these messages, and the results were published to Kafka Topic B.
• Successfully deployed on an Amazon EMR cluster, running smoothly.

New Build Overview:

• Transitioned to receiving Confluent AVRO messages from Kafka Topic A in the same application.
• Utilized ABRiS library functions (from_avro() and to_avro()) to deserialize AVRO to JSON and vice versa.
• The rest of the code remained unchanged.
• Successfully deployed on an Amazon EMR cluster, initially working fine for the first 20 to 30 minutes.

Error in New Build:

After the initial working period, the streaming application encountered the following error:

ERROR Client: Application diagnostics message: Application application_1696171462852_0173 failed 2 times due to AM Container for appattempt_1696171462852_0173_000002 exited with exitCode: 137
Failing this attempt.Diagnostics: Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137.
Killed by external signal

Investigation Steps:

• Referenced the AWS Knowledge Center article here for troubleshooting.
• Despite following the suggested steps, the issue persists.

Specific Queries:

• Seeking your input on the possibility of ABRiS library functions (from_avro() and to_avro()) causing the issue.
• Additionally, exploring other potential factors, such as message payload size, that might be contributing to the problem.

Your insights and suggestions on resolving this matter would be greatly appreciated.

[cerveada]

Abris doesn't really works with json, it converts from avro to spark DataFrame. So it could also be some issue with json.

Best would be to use some profile to see what is really happening in the memory.

One Idea I have is the registryConfig. Do your application uses avro with many different registryConfig maps? That could cause a memory leak since the registry clients are cached for each config.

[Vivek-Merugu]

ABRiS Library Usage:

• Yes, I am utilizing the ABRiS library function from_avro() to convert AVRO to Spark DataFrame and then transforming the Spark DataFrame to JSON messages and vice-versa.

RegistryConfig in Use:

• The application employs three distinct registryConfig maps.

Memory Monitoring:

• Certainly, I will implement memory monitoring and keep you updated.