Fix dependency on snapshot build of parquet-avro and spark.

[netflix]

Fix dependency on snapshot build of parquet-avro and spark.

[omalley]

I got it working with:

• parquet-mr (master 6e0cc729d)
• spark (branch-2.3 aa1764ba1a)

I'm not that familiar with gradle and probably misread the error message. I agree that core doesn't look like it has a dependency on parquet. :)

[omalley]

Also, which git hash should I be using for spark? The tip of branch-2.3 (75131ee867bc) didn't work, even after I changed build.gradle to use spark 2.3.1-SNAPSHOT. sigh

[rdblue]

Core shouldn't have a dependency on Parquet, I was trying to keep that contained to just iceberg-parquet. I'll have a look.

Spark's DataSourceV2 implementation has been changing, so I've been avoiding an update... for example, lots of the classes were just recently renamed in time for the next RC. I plan to base this on 2.3.0 when it is released.

[omalley]

Can you tell me which hash you use for spark?

[rdblue]

version=2.3.0-SNAPSHOT
user=blue
revision=aa1764ba1addbe7ad79344d5640bf6426267a38c
branch=master
date=2017-12-13T20:22:53Z
url=https://github.com/rdblue/spark.git

[omalley]

Thanks, Ryan. That did the trick.

[rdblue]

Do you know how iceberg-core depends on parquet-avro? I'm looking at the dependencies and I don't see it:

gradle iceberg-core:dependencies --configuration compile

compile - Dependencies for source set 'main' (deprecated, use 'implementation ' instead).                                                                      
+--- project :iceberg-api              
+--- project :iceberg-avro             
|    +--- project :iceberg-api         
|    +--- project :iceberg-common      
|    \--- org.apache.avro:avro:1.8.2   
|         +--- org.codehaus.jackson:jackson-core-asl:1.9.13                    
|         +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13                  
|         |    \--- org.codehaus.jackson:jackson-core-asl:1.9.13               
|         +--- com.thoughtworks.paranamer:paranamer:2.7                        
|         +--- org.xerial.snappy:snappy-java:1.1.1.3                           
|         +--- org.apache.commons:commons-compress:1.8.1                       
|         +--- org.tukaani:xz:1.5      
|         \--- org.slf4j:slf4j-api:1.7.7                                       
+--- com.fasterxml.jackson.core:jackson-databind:2.6.7                         
|    +--- com.fasterxml.jackson.core:jackson-annotations:2.6.0                 
|    \--- com.fasterxml.jackson.core:jackson-core:2.6.7                        
\--- com.fasterxml.jackson.core:jackson-core:2.6.7                             

Iceberg's Spark support does depend on Parquet, which I agree sucks for now. We'll get a Parquet release out to fix this.

[rdblue]

You might be able to get at least core working with the latest parquet-avro release. It won't work with column ids, but it might compile and unblock you guys for Avro and ORC.

[rdblue]

#16 updated Spark to the 2.3.0 release. Thanks, @julienledem!

We'll get a Parquet release out soon, too.

[rdblue]

Just updated to Parquet 1.10.0.