Comments (3)
So this would be similar to the behavior of spark.hadoop
properties?
I think that would make sense. We want to configure writers primarily through Iceberg table configuration, and it would be good to have a way to do that other than just a small list of known and supported properties.
from iceberg.
Originally I was thinking we could put it in sparkSession.write.option(iceberg.spark.hadoop.<opt>)
but now that you mention it this could make more sense to be in the Table's inherent properties. I can put up a patch for this.
from iceberg.
We want configuration to come from multiple places. A write may want to use a different format, so that would be passed like your example. But most things we want to apply at a table level and not at a job level. Parquet tuning parameters are a good example of that -- we want to set them once and know that all writers know what to do.
A PR would be great. Thanks!
from iceberg.
Related Issues (20)
- Allow Specifying Partitioning Function for External Mappings HOT 1
- Document Spark Issues Affecting the Iceberg Implementation
- Snapshot-Level Metrics and Statistics
- Cannot convert unknown primitive type: required int96 `timestamp` HOT 2
- Add More Documentation HOT 4
- Custom metadata in data files
- Custom InputFile / OutputFile providers for Spark HOT 4
- Support for AWS Glue as an alternative Hive metastore implementation HOT 5
- .travis.yml: The 'sudo' tag is now deprecated in Travis CI
- Port to Azure or Google Cloud?
- Any way to integrate Iceberg with AWS EMR Glue metastore?
- Problem inserting data into a table with structs (iceberg-spark)
- java.lang.AbstractMethodError: Method org/apache/iceberg/spark/source/SparkTable.newWriteBuilder HOT 1
- Long wait time for retry when refreshing table metadata HOT 1
- Support Customizing The Location Of Data Files Written By The Spark Data Source HOT 3
- Upgrade to Spark 2.4.0 HOT 2
- iceberg-runtime jar on JitPack is empty HOT 4
- Snapshot cryptographic integrity HOT 1
- Replace Literals with Stateless Functions
- Predicate Compiler for Evaluators
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from iceberg.