Comments (2)
Hello! I, honestly, don't remember. I think it was difficult to update the index file when you add more files to it and you would have to reload the existing index if it is cached in memory, but I don't remember the exact reasons. Feel free to experiment and let me know what difficulties you encounter, maybe we can address those.
from parquet-index.
ok! If there are any questions, I'll ask you,So far I've basically implemented the append mode and I'm testing for accuracy
from parquet-index.
Related Issues (20)
- Is 0.5.0 coming any soon? HOT 2
- Add disk spill for dictionary filter
- Report original scheme in error message when no column found
- Support for Spark 2.1
- Is there support for spark 2.1 or 2.2 in the roadmap? HOT 1
- NullPointerException when running on Yarn HOT 13
- Can it work with Spark SQL directly? HOT 6
- Failed to merge incompatible data types LongType and StringType HOT 5
- Just curious, how is going on for this project HOT 8
- Using s3a for index metastore throws a "Wrong FS" exception HOT 4
- How about introduce a light filter strategy that based on the col used in Dataset#repartition. HOT 10
- Check if project still works on Spark 2.3 and Spark 2.4 HOT 2
- Handle Dictionary pages and build filters from those
- java.lang.IllegalArgumentException: Wrong FS: file:/usr/local/spark-2.2.0/bin/index_metastore/catalog/parquet/hdfs/user/gsbdc/dbdatas/test/fact_ord_order_test/part-f-00015, expected: hdfs://nameservice1 HOT 11
- BinaryType column is not supported (if string type not support to create index or length 32 is too long) HOT 3
- NoSuchMethodError on accessing data through index HOT 1
- how to read from HDFS multiple parquet files with spark.index.create .mode("overwrite").indexBy($"cellid").parquet HOT 13
- How to read a parquet file from multiple directories at once HOT 5
- Fix CI build and update master to build with Spark 3.0
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from parquet-index.