Sparkling Water assembly jar includes non-shaded AWS SDK dependencies

[h2oai]

Sparkling Water assembly jar includes non-shaded AWS SDK dependencies

[mdymczyk]

@jtornblad using different versions of such libraries might render H2O unusable (if there were changes in the APIs etc. that we're using) so I'm not sure if that's how we should do it by default.

Personally I prefer it as it is now AND only if you need to change a library version you should shadow it in your build script as it's the exception (and definitely not a rule).

That is if I understood you correctly: you want us to shadow all libs which would make users define them themselves.

[jtornblad]

@mdymczyk I believe that if Sparkling Water shaded its dependencies this would (a) keep them fully functional for its own internal support of S3 for instance and (b) allow end-user applications of which Sparkling Water is a dependency to include their own versions without worry of colliding with older versions in the Sparkling Water assembly.

If Sparkling Water needed to invoke S3 it would of course have its shaded version... if the user's application wanted to invoke S3 directly they would of course have to include whatever S3 SDK version they like... but this is normal of course.

Tried to find some guidelines out there on the interpipes for shading, especially when the assembly jar is itself going to be a dependency in another application (i.e., when it is being treated as a library)... not much to go on (except lots of grumbling of course about the shabby state of the problem related to the JVM, class loaders and library versions).

At least if Sparkling Water is not going to shade its dependencies in the assembly it would be developer-friendly to declare all the dependencies that are present and their versions along with a heads-up on what magical failures will happen if the user starts including any of their own older or newer version of those dependencies... i.e., complaints about method signatures not matching, the perhaps random nature of such failures depending on ordering of class loading operations, etc. ... would definitely help remove some of the mystery.

Sparkling Water now declares explicitly the version of Spark they are supporting... it would be great to be up-front about included other libraries (like AWS for S3 support) and state they are not shaded.

Any non-trivial Spark job including H2O and running in AWS and invoking any AWS API which shares lower-level dependencies such as http support, etc. is bound to have problems.

[mdymczyk]

Hey @jtornblad I do agree with a lot of your points, personally I'm not a fan of shading, though, as I had many problems with it when developing some plugins for Elasticsearch (which from what I know stopped shading its deps exactly because of such issues around version 2.0).

I'll have a chat with @mmalohlava and get back to you asap.

[jakubhava]

Hi @jtornblad , I create JIRA for the issue - https://0xdata.atlassian.net/browse/SW-418 , please use this link to track the state of the issue.