imduffy15/spark-avro-compactor

Spark job for compacting avro files together

spark-avro-compactor

Spark job for compacting avro files together

Running locally

1. local=true sbt
2. runMain ie.ianduffy.spark.avro.compactor.Runner -i inputPath --output outputPath --schema-registry-url https://confluent-schema-registry --schema-registry-subject schema-subject

Contributors

1. Kevin Eid
2. Peter Barron
3. Ian Duffy

spark-avro-compactor's Issues

[Feature] Allow time-based compaction

In case the avro files are already partioned by event time, it would be desirable to run compaction with an additional upper bound in terms of event time ti_max so that a single file will only be appended in case e_max -_t e_min_t <= ti_max.

In that case, the resultant files retain their time semantics.

Compression Ratio

Can you give some insights around how did you arrive at compression ratio for these formats?

Why not just using datafu?

Hello, thanks a lot for great work and the article, but I'm wondering why not just using DataFu's CombinedAvroKeyInputFormat?

It should solve the issue with number of spark tasks required to process the input, but listing of s3 objects will remain to be slow (but it may not be the problem as this job still has to read the input data for at least one time).

Serialization

Great job!
One question: Did you face any serialization issues when rdd.coalesce(numPartitions) and how was the performance?