Comments (5)
The problem happened because there wasn't enough memory allocated to the Spark executor. After increasing the Spark executor memory, the issue was resolved.
Hence, closing this issue.
from abris.
What makes you believe that this issue is caused by Abris?
from abris.
Thank you for your response. I appreciate your assistance in addressing this issue. Here's a breakdown of the situation:
Old Build Overview:
- The initial Spark Streaming Scala code processed JSON messages from Kafka Topic A.
- Operations were performed on these messages, and the results were published to Kafka Topic B.
- Successfully deployed on an Amazon EMR cluster, running smoothly.
New Build Overview:
- Transitioned to receiving Confluent AVRO messages from Kafka Topic A in the same application.
- Utilized ABRiS library functions (from_avro() and to_avro()) to deserialize AVRO to JSON and vice versa.
- The rest of the code remained unchanged.
- Successfully deployed on an Amazon EMR cluster, initially working fine for the first 20 to 30 minutes.
Error in New Build:
After the initial working period, the streaming application encountered the following error:
ERROR Client: Application diagnostics message: Application application_1696171462852_0173 failed 2 times due to AM Container for appattempt_1696171462852_0173_000002 exited with exitCode: 137
Failing this attempt.Diagnostics: Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137.
Killed by external signal
Investigation Steps:
- Referenced the AWS Knowledge Center article here for troubleshooting.
- Despite following the suggested steps, the issue persists.
Specific Queries:
- Seeking your input on the possibility of ABRiS library functions (from_avro() and to_avro()) causing the issue.
- Additionally, exploring other potential factors, such as message payload size, that might be contributing to the problem.
Your insights and suggestions on resolving this matter would be greatly appreciated.
from abris.
Abris doesn't really works with json, it converts from avro to spark DataFrame. So it could also be some issue with json.
Best would be to use some profile to see what is really happening in the memory.
One Idea I have is the registryConfig
. Do your application uses avro with many different registryConfig
maps? That could cause a memory leak since the registry clients are cached for each config.
from abris.
ABRiS Library Usage:
- Yes, I am utilizing the ABRiS library function from_avro() to convert AVRO to Spark DataFrame and then transforming the Spark DataFrame to JSON messages and vice-versa.
RegistryConfig in Use:
- The application employs three distinct registryConfig maps.
Memory Monitoring:
- Certainly, I will implement memory monitoring and keep you updated.
from abris.
Related Issues (20)
- from_avro converts `\uFFFD` to a question mark HOT 1
- schema registry being called with http instead of https HOT 2
- Improve code-coverage & add GH check action HOT 1
- Fix JaCoCo CI for PRs from forked repos HOT 1
- update madrapps/jacoco-report
- Detect different schema versions in batch HOT 5
- Revert pull_request action back HOT 3
- TopicNameStrategy issue HOT 1
- Split GitHub actions for tests and test coverage
- Multiple schemas in one topic example HOT 1
- Spark 3.4 Support HOT 13
- malformed records to topic HOT 2
- foreach batch download by schem id HOT 3
- Issues running inside Scala notebook on databricks HOT 1
- Fix tests for Spark 3.5.0
- Fix NoSuchMethodException in Spark 3.5.x
- get key from avro message HOT 3
- Compatibility with Spark 3.5 HOT 3
- Version 6.4.0 failing for Spark 3.5.0 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abris.