klangner/datafusion-orc

Implementation of Apache ORC file format use Apache Arrow in-memory format

orc-rust

A native Rust implementation of the Apache ORC file format, providing API's to read data into Apache Arrow in-memory arrays.

See the documentation for examples on how to use this crate.

Supported features

This crate currently only supports reading ORC files into Arrow arrays. Write support is planned (see Roadmap). The below features listed relate only to reading ORC files. At this time, we aim to support the ORCv1 specification only.

• Read synchronously & asynchronously (using Tokio)
• All compression types (Zlib, Snappy, Lzo, Lz4, Zstd)
• All ORC data types
• All encodings
• Rudimentary support for retrieving statistics
• Retrieving user metadata into Arrow schema metadata

Roadmap

The long term vision for this crate is to be feature complete enough to be donated to the arrow-rs project.

The following lists the rough roadmap for features to be implemented, from highest to lowest priority.

• Performance enhancements
• DataFusion integration
• Predicate pushdown
• Row indices
• Bloom filters
• Write from Arrow arrays
• Encryption

A non-Arrow API interface is not planned at the moment. Feel free to raise an issue if there is such a use case.

Version compatibility

No guarantees are provided about stability across versions. We will endeavour to keep the top level API's (ArrowReader and ArrowStreamReader) as stable as we can, but other API's provided may change as we explore the interface we want the library to expose.

Versions will be released on an ad-hoc basis (with no fixed schedule).

Mapping ORC types to Arrow types

The following table lists how ORC data types are read into Arrow data types:

ORC Data Type	Arrow Data Type
Boolean	Boolean
TinyInt	Int8
SmallInt	Int16
Int	Int32
BigInt	Int64
Float	Float32
Double	Float64
String	Utf8
Char	Utf8
VarChar	Utf8
Binary	Binary
Decimal	Decimal128
Date	Date32
Timestamp	Timestamp(Nanosecond, None)
Timestamp instant	Timestamp(Nanosecond, UTC)
Struct	Struct
List	List
Map	Map
Union	Union(_, Sparse)*

*Currently only supports a maximum of 127 variants

Contributing

All contributions are welcome! Feel free to raise an issue if you have a feature request, bug report, or a question. Feel free to raise a Pull Request without raising an issue first, as long as the Pull Request is descriptive enough.

Some tools we use in addition to the standard cargo that require installation are:

• taplo
• typos

cargo install typos-cli
cargo install taplo-cli

# Building the crate
cargo build

# Running the test suite
cargo test

# Simple benchmarks
cargo bench

# Formatting TOML files
taplo format

# Detect any typos in the codebase
typos

To regenerate/update the proto.rs file, execute the regen.sh script.

./regen.sh