Comments (2)
hn5092
commented on August 12, 2024
Root Cause: 构建的时候会复用dataframe, 因为spark plan会生成一些broadcast,我们之前优化qps的时候会去删除这些broadcast,导致再次使用的时候会报错,这里还有个疑点就是为什么要挂了才会触发,
Fix Design evidence: 使用小内存构建大的任务,导致其被yarn kill了之后重新运行能通过
Dev test evidence, Test cover by UT/IT ? it 无法覆盖
QA needed ? (Y/N) N
Test suggestions (other feature affected) to QA
from spark.
hn5092
commented on August 12, 2024
ke issue https://github.com/Kyligence/KAP/issues/9792
from spark.
Related Issues (20)
- 当没有yarn admin权限的时候,防止spark无法启动 HOT 1
- option to disable AmIpFilter
- broadcast join oom
- S3AFileSystem read retry before getFileStatus
- sql app status add task skewness detection
- standalone cluster manager
- define spark default db
- Add a static config to control cache size for generated classes
- upgrade jackson version
- fix AE remove root ShuffleExchangeExec which generated by RepartitionByExpression
- 暴露接口可以直接增加 executor/node 的黑名单
- spark-sql支持limit offset
- 优化parquet读取性能
- handle date skew HOT 1
- 把 parquet metric 展示到 spark ui 上 HOT 1
- show hive table privileges
- bug fix taskmetrics
- 如果有limit的情况下,根据source row count来判断是否一次取数
- 在某些场景下,org.apache.spark.util.ClosureCleaner#getClassReader会导致短暂几百毫秒延迟
- 提供配置,让用户决定是否优先用本地的hadoop conf HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark.