Is there any actual conversion implementation for arrow and parquet?

[chenyuanxing]

Is there any actual conversion implementation for arrow and parquet?

[wgtmac]

I think conversion between parquet and arrow is a valid use case. The parquet-java provides built-in row-level interfaces like avro/thrift/protobuf. Other parquet (Java) implementations (Presto/Trino/Spark) simply leverage the page & metadata reader/writer from this library to build extensions. Extending native arrow support would be a welcome extension to this library, IMO.

[wgtmac]

If possible to use C++, I think parquet-cpp in the Apache Arrow is the best solution to your case: https://arrow.apache.org/docs/cpp/parquet.html

[chenyuanxing]

Yes, We know there is a c++ implementation here, but I was wondering if there is a corresponding implementation for java, since all our code is java .

[chenyuanxing]

parquet-arrow The library looks like it's meant to do this, But I don't know why it's always just the schema part.

[chenyuanxing]

So, the library parquet-arrow hasn't been used yet? because it only has schema mappings.

And We've looked at transformations in Spark, which are missing some types due to limitations in Spark, such as uint.So it's not really a universal conversion.