imduffy15/spark-avro-compactor

Spark job for compacting avro files together

spark-avro-compactor's Issues

Compression Ratio

Can you give some insights around how did you arrive at compression ratio for these formats?

Why not just using datafu?

Hello, thanks a lot for great work and the article, but I'm wondering why not just using DataFu's CombinedAvroKeyInputFormat?

It should solve the issue with number of spark tasks required to process the input, but listing of s3 objects will remain to be slow (but it may not be the problem as this job still has to read the input data for at least one time).

[Feature] Allow time-based compaction

In case the avro files are already partioned by event time, it would be desirable to run compaction with an additional upper bound in terms of event time ti_max so that a single file will only be appended in case e_max -_t e_min_t <= ti_max.

In that case, the resultant files retain their time semantics.

Serialization

Great job!
One question: Did you face any serialization issues when rdd.coalesce(numPartitions) and how was the performance?