archivesunleashed/aut

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.

The Archives Unleashed Toolkit

The Archives Unleashed Toolkit is an open-source platform for analyzing web archives using Apache Spark, and makes use of Sparkling for parsing W/ARC records. The toolkit provides powerful tools for analytics and data processing. It is part of the Archives Unleashed Project.

To learn more about the Toolkit and how to use, please see our comprehensive documentation.

If you would like a more in-depth look at the project, please check out the following two articles:

• Nick Ruest, Jimmy Lin, Ian Milligan, and Samantha Fritz. The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives. Proceedings of the 2020 IEEE/ACM Joint Conference on Digital Libraries (JCDL 2020), Wuhan, China.
• Jimmy Lin, Ian Milligan, Jeremy Wiebe, and Alice Zhou. Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives. ACM Journal on Computing and Cultural Heritage, 10(4), Article 22, 2017.

Dependencies

• Java 11
• Python 3.7.3+ (PySpark)
• Scala 2.12+
• Apache Spark (Hadoop 2.7) 3.0.3+

More information on setting up dependencies can be found here.

Building

Clone the repo:

git clone http://github.com/archivesunleashed/aut.git

You can then build The Archives Unleashed Toolkit.

mvn clean install

Usage

The Toolkit can be used to submit a variety of extraction jobs with spark-submit, as well used as a library via spark-submit, pyspark, or in your own application. More information on using the Toolkit can be found here.

Citing Archives Unleashed

How to cite the Archives Unleashed Toolkit or Cloud in your research:

Nick Ruest, Jimmy Lin, Ian Milligan, and Samantha Fritz. 2020. The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 (JCDL '20). Association for Computing Machinery, New York, NY, USA, 157–166. DOI:https://doi.org/10.1145/3383583.3398513

Your citations help to further the recognition of using open-source tools for scientific inquiry, assists in growing the web archiving community, and acknowledges the efforts of contributors to this project.

License

Licensed under the Apache License, Version 2.0.

Acknowledgments

This work is primarily supported by the Andrew W. Mellon Foundation. Other financial and in-kind support comes from the Social Sciences and Humanities Research Council, Compute Canada, the Ontario Ministry of Research, Innovation, and Science, York University Libraries, Start Smart Labs, and the Faculty of Arts and David R. Cheriton School of Computer Science at the University of Waterloo.

Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.

aut's Issues

Get a project setup on sonatype

Create tests for ExtractBoilerpipeText.scala

ExtractBoilerpipeText.scala has no test coverage. We need to create tests for it.

Improve TupleFormatter.scala test coverage

TupleFormatter.scala test coverage can be improved.

What do we do with all the documentation at docs.warcbase.org?

Create tests for ComputeMD5.scala

ComputeMD5.scala has no test coverage. We need to create tests for it.

Unnecessary buffer copying

From TODO in src/main/java/io/archivesunleashed/data/WarcRecordUtils.java and src/main/java/io/archivesunleashed/data/ArcRecordUtils.java that will be removed with the resolution of #3:

These methods work fine, but there's a lot of unnecessary buffer copying, which is terrible from a performance perspective.

Create tests for ExtractPopularImages.scala

ExtractPopularImages.scala has no test coverage. We need to create tests for it.

Create tests for ExtractGraph.scala

ExtractGraph.scala has no test coverage. We need to create tests for it.

Retarget Hadoop

Maven currently targets CDH 5.7.1 for Hadoop/Spark related artifacts. This should probably be modified since it was pegged against the Hadoop version running at the Maryland cluster.

My suggestion is to peg to whatever version Altiscale is running, since that will likely become our canonical processing cluster moving forward.

Create tests for JsonUtil.scala

JsonUtil.scala has no test coverage. We need to create tests for it.

Improve RecordLoader.scala test coverage

RecordLoader.scala test coverage can be improved.

Create tests for ExtractTextFromPDFs.scala

ExtractTextFromPDFs.scala has no test coverage. We need to create tests for it.

Setup Victims

Setup codecov.io

Setup codecov.io on this repo.

See this example: https://github.com/codecov/example-java-maven

Improve ExtractDate.scala test coverage

ExtractDate.scala test coverage can be improved.

Improve ExtractDomain.scala test coverage

ExtractDomain.scala test coverage can be improved.

Investigate Jupyter notebooks on Altiscale

Guide here: https://documentation.altiscale.com/jupyter

Update Scala version

Blocks #32

[ERROR] scalac error: 'jvm-1.8' is not a valid choice for '-target'

We need to update the Scala version. We're using 2.10.5 which is not available with Java 8.

Improve RecordRDD.scala test coverage

RecordRDD.scala test coverage can be improved.

More work needs to be done on the pom.xml to get us to a release.

Follow-up to #7 and #24

[INFO] 
[INFO] --- maven-site-plugin:3.3:deploy (default-deploy) @ aut ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:53 min
[INFO] Finished at: 2017-08-24T13:37:42-04:00
[INFO] Final Memory: 54M/1667M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.3:deploy (default-deploy) on project aut: Wagon protocol '' doesn't support directory copying -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

I didn't fully configure the GitHub pages plugin, and we're missing some more release plugins to get us to Sonatype correctly.

Visualization Repo (aut-viz)

We decided to move the Warcbase visualization content over to a new repository (aut-viz). This would include:

• https://github.com/lintool/warcbase/tree/master/vis
• https://github.com/lintool/warcbase/tree/master/warcbase-core/src/main/python

There's cleanup to do once it is moved over to aut-viz with history, which I can take care of. But @ruebot has kindly offered to migrate and keep history with his GitHub skills (then he can assign me!).

Investigate our test coverage according to codecov.io

Double check annotations for test coverage; make sure things are setup right.

Once we're good with the above, create sub-tickets for test coverage.

Setup license headers and mycila

What do we do with the existing license headers that mention warcbase? Also, do we need a header for the LICENSE file? If so, should we add each of our universities, or pull in the entire history of the project, or just our names?

Upgrade to Spark 2 on Altiscale

Let's upgrade to Spark 2 on Altiscale, following documentation here.

Create tests for NER3Classifier.scala

NER3Classifier.scala has no test coverage. We need to create tests for it.

Is src/main/java/io/archivesunleashed/demo required?

https://github.com/archivesunleashed/aut/tree/master/src/main/java/io/archivesunleashed/demo

Create tests for WriteGDF.scala

WriteGDF.scala has no test coverage. We need to create tests for it.

Setup to publish javadocs on ghpages

mvn site-deploy -DskipTests is still failing

Maybe? https://stackoverflow.com/questions/26144541/why-am-i-getting-maven-error-unable-to-determine-if-resource-x-exists-in-http

[ERROR] Unable to determine if resource xml-apis:xml-apis:jar:1.4.01:compile exists in http://maven.glassfish.org/content/groups/glassfish
[ERROR] Unable to determine if resource xmlenc:xmlenc:jar:0.52:compile exists in http://maven.glassfish.org/content/groups/glassfish
[ERROR] Unable to determine if resource xpp3:xpp3_min:jar:1.1.3.4.O:compile exists in http://maven.glassfish.org/content/groups/glassfish
[INFO] Generating "Dependency Convergence" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Dependency Information" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Distribution Management" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "About" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Issue Management" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Licenses" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Plugin Management" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Plugins" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Team" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Source Code Management" report    --- maven-project-info-reports-plugin:2.9
[INFO] Generating "Summary" report    --- maven-project-info-reports-plugin:2.9
[INFO] 
[INFO] --- maven-javadoc-plugin:2.10.3:jar (attach-javadocs) @ aut ---
[INFO] Building jar: /tmp/aut/target/aut-0.9.0-javadoc.jar
[INFO] 
[INFO] >>> maven-javadoc-plugin:2.10.3:javadoc (attach-javadocs) > generate-sources @ aut >>>
[INFO] 
[INFO] <<< maven-javadoc-plugin:2.10.3:javadoc (attach-javadocs) < generate-sources @ aut <<<
[INFO] 
[INFO] --- maven-javadoc-plugin:2.10.3:javadoc (attach-javadocs) @ aut ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:deploy (default-deploy) @ aut ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 16:33 min
[INFO] Finished at: 2017-08-24T16:26:39-04:00
[INFO] Final Memory: 60M/1671M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.3:deploy (default-deploy) on project aut: Wagon protocol '' doesn't support directory copying -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Create tests for TweetUtils.scala

TweetUtils.scala has no test coverage. We need to create tests for it.

Native notebook support

Firing up a Juypter notebook with AUT should be easy!

Remove `src/main/python`

Setup checkstyle

Improve StringUtils.scala test coverage

StringUtils.scala test coverage can be improved.

Migrate warcbase-resources to aut-resources

We have resources stored over in the warcbase-resources repository. We should have it migrated to a new aut-resources repository under the archivesunleashed GitHub organization.

Currently @lintool has admin permission on warcbase-resources.

Improve ExtractLinks.scala test coverage

ExtractLinks.scala test coverage can be improved.

Improve ExtractImageLinks.scala test coverage

ExtractImageLinks.scala test coverage can be improved.

Create tests for NERCombinedJson.scala

NERCombinedJson.scala has no test coverage. We need to create tests for it.

Remove org.apache.commons.httpclient

Commons HttpClient is EOL:

"The Commons HttpClient project is now end of life, and is no longer being developed. It has been replaced by the Apache HttpComponents project in its HttpClient and HttpCore modules, which offer better performance and more flexibility."

We make use of it in src/main/java/io/archivesunleashed/data/WarcRecordUtils.java.

Specifically:

• aut/src/main/java/io/archivesunleashed/data/WarcRecordUtils.java

  Line 28 in 3fb85e4

  import org.apache.commons.httpclient.HttpParser;

• aut/src/main/java/io/archivesunleashed/data/WarcRecordUtils.java

  Lines 133 to 153 in 3fb85e4

  	/**
  	* Extracts contents of the body from a {@code WARCRecord}.
  	* Excludes HTTP headers.
  	*
  	* @param record the {@code WARCRecord}
  	* @return contents of the body
  	* @throws IOException if there is an issue
  	*/
  	public static byte[] getBodyContent(final WARCRecord record)
  	throws IOException {
  	ByteArrayOutputStream baos = new ByteArrayOutputStream();
  	String line = HttpParser.readLine(record, WARC_HEADER_ENCODING);
  	if (line == null) {
  	return null;
  	}
  	// Just using parseHeaders to move down input stream to body
  	HttpParser.parseHeaders(record, WARC_HEADER_ENCODING);
  	record.dump(baos);
  	return baos.toByteArray();
  	}

We can probably replace usage with this.

See also:

• #5
• iipc/webarchive-commons#78

NER breaks for WARC files?

Hello,

I've been trying to use the NER module, described here: http://docs.archivesunleashed.io/Spark-Named-Entity-Recognition/, for WARC files and I haven't been successful so far. I tested the NER module against ARC files and it works correctly as shown in the image below:

The error message I receive when trying to use WARC file is as follows:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: After retry (Offset 0)
	at org.archive.io.ArchiveReader$ArchiveRecordIterator.next(ArchiveReader.java:512)
	at org.archive.io.ArchiveReader$ArchiveRecordIterator.next(ArchiveReader.java:436)
	at io.archivesunleashed.mapreduce.WacArcInputFormat$ArcRecordReader.nextKeyValue(WacArcInputFormat.java:166)
	at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:168)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply$mcV$sp(PairRDDFunctions.scala:1195)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply(PairRDDFunctions.scala:1195)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply(PairRDDFunctions.scala:1195)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1277)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1203)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1183)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
] (Offset 0).
	at org.archive.io.arc.ARCRecord.getTokenizedHeaderLine(ARCRecord.java:341)
	at org.archive.io.arc.ARCRecord.parseHeaders(ARCRecord.java:243)
	at org.archive.io.arc.ARCRecord.<init>(ARCRecord.java:206)
	at org.archive.io.arc.ARCReader.createArchiveRecord(ARCReader.java:149)
	at org.archive.io.arc.ARCReaderFactory$CompressedARCReader$1.innerNext(ARCReaderFactory.java:426)
	at org.archive.io.ArchiveReader$ArchiveRecordIterator.exceptionNext(ArchiveReader.java:537)
	at org.archive.io.ArchiveReader$ArchiveRecordIterator.next(ArchiveReader.java:505)
	... 20 more

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1922)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1209)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1154)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1154)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1154)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1060)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1026)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1026)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1026)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:952)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:952)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:952)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:951)
	at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1457)
	at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1436)
	at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1436)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
	at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1436)
	at io.archivesunleashed.spark.matchbox.ExtractEntities$.extractAndOutput(ExtractEntities.scala:66)
	at io.archivesunleashed.spark.matchbox.ExtractEntities$.extractFromRecords(ExtractEntities.scala:35)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:63)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:73)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:75)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:77)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:79)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:81)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:83)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:85)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:87)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:89)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:91)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:93)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:95)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:97)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:99)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:101)
	at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:103)
	at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:105)
	at $iwC$$iwC$$iwC$$iwC.<init>(<console>:107)
	at $iwC$$iwC$$iwC.<init>(<console>:109)
	at $iwC$$iwC.<init>(<console>:111)
	at $iwC.<init>(<console>:113)
	at <init>(<console>:115)
	at .<init>(<console>:119)
	at .<clinit>(<console>)
	at .<init>(<console>:7)
	at .<clinit>(<console>)
	at $print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
	at notebook.kernel.Repl$$anonfun$6.apply(Repl.scala:202)
	at notebook.kernel.Repl$$anonfun$6.apply(Repl.scala:202)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
	at scala.Console$.withOut(Console.scala:126)
	at notebook.kernel.Repl.evaluate(Repl.scala:201)
	at notebook.client.ReplCalculator$$anonfun$10$$anon$1$$anonfun$27.replEvaluate$1(ReplCalculator.scala:402)
	at notebook.client.ReplCalculator$$anonfun$10$$anon$1$$anonfun$27.apply(ReplCalculator.scala:415)
	at notebook.client.ReplCalculator$$anonfun$10$$anon$1$$anonfun$27.apply(ReplCalculator.scala:396)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.RuntimeException: After retry (Offset 0)
	at org.archive.io.ArchiveReader$ArchiveRecordIterator.next(ArchiveReader.java:512)
	at org.archive.io.ArchiveReader$ArchiveRecordIterator.next(ArchiveReader.java:436)
	at io.archivesunleashed.mapreduce.WacArcInputFormat$ArcRecordReader.nextKeyValue(WacArcInputFormat.java:166)
	at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:168)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply$mcV$sp(PairRDDFunctions.scala:1195)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply(PairRDDFunctions.scala:1195)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply(PairRDDFunctions.scala:1195)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1277)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1203)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1183)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
] (Offset 0).
	at org.archive.io.arc.ARCRecord.getTokenizedHeaderLine(ARCRecord.java:341)
	at org.archive.io.arc.ARCRecord.parseHeaders(ARCRecord.java:243)
	at org.archive.io.arc.ARCRecord.<init>(ARCRecord.java:206)
	at org.archive.io.arc.ARCReader.createArchiveRecord(ARCReader.java:149)
	at org.archive.io.arc.ARCReaderFactory$CompressedARCReader$1.innerNext(ARCReaderFactory.java:426)
	at org.archive.io.ArchiveReader$ArchiveRecordIterator.exceptionNext(ArchiveReader.java:537)
	at org.archive.io.ArchiveReader$ArchiveRecordIterator.next(ArchiveReader.java:505)
	... 20 more

I tested this against AUT's official docker https://github.com/archivesunleashed/docker-aut in a spark notebook. I did not attempt it in spark-shell.

Create tests for ExtractEntities.scala

ExtractEntities.scala has no test coverage. We need to create tests for it.

PySpark support

We need PySpark support.

Update to use Java 8

Improve RemoveHTML.scala test coverage

RemoveHTML.scala test coverage can be improved.

DataFrames support

We need DataFrames support. Not sure if the best course of action is to resolve #12 and build PySpark DataFrames directly, or to build DataFrame support and hash out in Scala before making sure everything works in PySpark.

Improve GenericArchiveRecord.scala test coverage

GenericArchiveRecord.scala test coverage can be improved.

Do we want to enable DependencyConvergence in maven-enforcer?

Do we want to enable this?

If we do, we have to sort this out:

[WARNING] 
Dependency convergence error for com.google.guava:guava:11.0.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.curator:curator-client:2.7.1
          +-com.google.guava:guava:16.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.apache.hadoop:hadoop-yarn-client:2.6.0-cdh5.7.1
            +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.apache.hadoop:hadoop-yarn-server-common:2.6.0-cdh5.7.1
            +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-yarn-api:2.6.0-cdh5.7.1
        +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.curator:curator-recipes:2.7.1
      +-org.apache.curator:curator-framework:2.7.1
        +-com.google.guava:guava:16.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.curator:curator-recipes:2.7.1
      +-com.google.guava:guava:16.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-hdfs:0.8.2
        +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-s3:0.8.2
        +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-local:0.8.2
        +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.intel.chimera:chimera:0.9.2
      +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-com.google.guava:guava:17.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-tl.lin:lintools-datatypes:1.0.0
    +-com.google.guava:guava:18.0

[WARNING] 
Dependency convergence error for javax.xml.bind:jaxb-api:2.2.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-javax.xml.bind:jaxb-api:2.2.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-de.jollyday:jollyday:0.4.7
      +-javax.xml.bind:jaxb-api:2.2.7

[WARNING] 
Dependency convergence error for commons-lang:commons-lang:2.6 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.apache.hadoop:hadoop-yarn-client:2.6.0-cdh5.7.1
            +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-yarn-api:2.6.0-cdh5.7.1
        +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-commons-lang:commons-lang:2.4

[WARNING] 
Dependency convergence error for org.apache.httpcomponents:httpcore:4.1.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-net.java.dev.jets3t:jets3t:0.9.0
      +-org.apache.httpcomponents:httpcore:4.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-org.apache.httpcomponents:httpcore:4.2.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:httpservices:4.5.5
      +-org.apache.httpcomponents:httpcore:4.2.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:httpservices:4.5.5
      +-org.apache.httpcomponents:httpmime:4.2.6
        +-org.apache.httpcomponents:httpcore:4.2.5

[WARNING] 
Dependency convergence error for org.xerial.snappy:snappy-java:1.0.4.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro:1.7.6-cdh5.7.1
        +-org.xerial.snappy:snappy-java:1.0.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.xerial.snappy:snappy-java:1.0.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.xerial.snappy:snappy-java:1.0.5

[WARNING] 
Dependency convergence error for xml-apis:xml-apis:1.3.03 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-com.io7m.xom:xom:1.2.10
      +-xml-apis:xml-apis:1.3.03
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-com.io7m.xom:xom:1.2.10
      +-xalan:xalan:2.7.0
        +-xml-apis:xml-apis:1.0.b2

[WARNING] 
Dependency convergence error for org.apache.commons:commons-lang3:3.3.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.commons:commons-lang3:3.3.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.apache.commons:commons-lang3:3.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-hdfs:0.8.2
        +-org.apache.commons:commons-lang3:3.0

[WARNING] 
Dependency convergence error for com.google.code.gson:gson:2.2.4 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-com.google.code.gson:gson:2.2.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-tl.lin:lintools-datatypes:1.0.0
    +-com.google.code.gson:gson:2.3.1

[WARNING] 
Dependency convergence error for org.apache.tika:tika-core:1.9 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-core:1.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.tika:tika-core:1.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.gagravarr:vorbis-java-tika:0.6
      +-org.apache.tika:tika-core:1.5

[WARNING] 
Dependency convergence error for io.netty:netty:3.6.2.Final paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-io.netty:netty:3.6.2.Final
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf
      +-io.netty:netty:3.6.6.Final

[WARNING] 
Dependency convergence error for commons-codec:commons-codec:1.4 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-net.java.dev.jets3t:jets3t:0.9.0
      +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-commons-codec:commons-codec:1.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.poi:poi:3.12
      +-commons-codec:commons-codec:1.9

[WARNING] 
Dependency convergence error for com.thoughtworks.paranamer:paranamer:2.3 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro:1.7.6-cdh5.7.1
        +-com.thoughtworks.paranamer:paranamer:2.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-com.thoughtworks.paranamer:paranamer:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.module:jackson-module-scala_2.10:2.2.3
      +-com.thoughtworks.paranamer:paranamer:2.3

[WARNING] 
Dependency convergence error for commons-logging:commons-logging:1.1.3 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.intel.chimera:chimera:0.9.2
      +-commons-logging:commons-logging:1.1.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.springframework:spring-core:3.0.6.RELEASE
      +-commons-logging:commons-logging:1.1.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.pdfbox:pdfbox:1.8.9
      +-org.apache.pdfbox:fontbox:1.8.9
        +-commons-logging:commons-logging:1.1.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.pdfbox:pdfbox:1.8.9
      +-commons-logging:commons-logging:1.1.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-com.github.junrar:junrar:0.7
      +-org.apache.commons:commons-vfs2:2.0
        +-commons-logging:commons-logging:1.1.1

[WARNING] 
Dependency convergence error for commons-net:commons-net:3.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-commons-net:commons-net:3.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-commons-net:commons-net:2.2

[WARNING] 
Dependency convergence error for org.scala-lang:scala-library:2.10.5 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.scalatest:scalatest_2.10:2.2.5
    +-org.scala-lang:scala-library:2.10.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.scala-lang:scala-library:2.10.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-com.chuusai:shapeless_2.10.5:2.0.0
    +-org.scala-lang:scala-library:2.10.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.twitter:chill_2.10:0.5.0
      +-org.scala-lang:scala-library:2.10.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf
      +-org.spark-project.akka:akka-actor_2.10:2.2.3-shaded-protobuf
        +-org.scala-lang:scala-library:2.10.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf
      +-org.scala-lang:scala-library:2.10.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-slf4j_2.10:2.2.3-shaded-protobuf
      +-org.scala-lang:scala-library:2.10.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.scala-lang:scala-library:2.10.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-org.json4s:json4s-ast_2.10:3.2.10
          +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-org.scala-lang:scalap:2.10.0
          +-org.scala-lang:scala-compiler:2.10.0
            +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-org.scala-lang:scalap:2.10.0
          +-org.scala-lang:scala-compiler:2.10.0
            +-org.scala-lang:scala-reflect:2.10.5
              +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.module:jackson-module-scala_2.10:2.2.3
      +-org.scala-lang:scala-library:2.10.2

[WARNING] 
Dependency convergence error for org.apache.zookeeper:zookeeper:3.4.5-cdh5.7.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.zookeeper:zookeeper:3.4.5-cdh5.7.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.curator:curator-client:2.7.1
          +-org.apache.zookeeper:zookeeper:3.4.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.zookeeper:zookeeper:3.4.5-cdh5.7.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.apache.hadoop:hadoop-yarn-server-common:2.6.0-cdh5.7.1
            +-org.apache.zookeeper:zookeeper:3.4.5-cdh5.7.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.curator:curator-recipes:2.7.1
      +-org.apache.curator:curator-framework:2.7.1
        +-org.apache.zookeeper:zookeeper:3.4.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.curator:curator-recipes:2.7.1
      +-org.apache.zookeeper:zookeeper:3.4.6

[WARNING] 
Dependency convergence error for xerces:xercesImpl:2.9.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-xerces:xercesImpl:2.9.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-com.io7m.xom:xom:1.2.10
      +-xerces:xercesImpl:2.8.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-com.syncthemall:boilerpipe:1.2.2
    +-net.sourceforge.nekohtml:nekohtml:1.9.20
      +-xerces:xercesImpl:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-com.syncthemall:boilerpipe:1.2.2
    +-xerces:xercesImpl:2.11.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-xerces:xercesImpl:2.11.0

[WARNING] 
Dependency convergence error for com.google.code.findbugs:jsr305:3.0.0 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-com.google.code.findbugs:jsr305:3.0.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.spark:spark-network-common_2.10:1.6.0-cdh5.7.1
      +-com.google.code.findbugs:jsr305:1.3.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.spark:spark-unsafe_2.10:1.6.0-cdh5.7.1
      +-com.google.code.findbugs:jsr305:1.3.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.google.code.findbugs:jsr305:1.3.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.module:jackson-module-scala_2.10:2.2.3
      +-com.google.code.findbugs:jsr305:2.0.1

[WARNING] 
Dependency convergence error for org.apache.commons:commons-compress:1.4.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro:1.7.6-cdh5.7.1
        +-org.apache.commons:commons-compress:1.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.commons:commons-compress:1.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-org.apache.commons:commons-compress:1.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.commons:commons-compress:1.9

[WARNING] 
Dependency convergence error for junit:junit:4.12 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-junit:junit:4.12
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.openwayback:openwayback-access-control-core:1.0.2
      +-junit:junit:4.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.openwayback:openwayback-access-control-core:1.0.2
      +-org.codehaus.jettison:jettison:1.0-beta-1
        +-junit:junit:3.8.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-com.googlecode.json-simple:json-simple:1.1.1
      +-junit:junit:4.10

[WARNING] 
Dependency convergence error for org.apache.httpcomponents:httpclient:4.2.5 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.httpcomponents:httpclient:4.2.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-aws:2.6.0-cdh5.7.1
        +-com.amazonaws:aws-java-sdk-s3:1.10.6
          +-com.amazonaws:aws-java-sdk-core:1.10.6
            +-org.apache.httpcomponents:httpclient:4.3.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-net.java.dev.jets3t:jets3t:0.9.0
      +-org.apache.httpcomponents:httpclient:4.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:httpservices:4.5.5
      +-org.apache.httpcomponents:httpclient:4.2.6

[WARNING] 
Dependency convergence error for org.jsoup:jsoup:1.7.3 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.jsoup:jsoup:1.7.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:grib:4.5.5
      +-org.jsoup:jsoup:1.7.2

[WARNING] 
Dependency convergence error for org.netpreserve.commons:webarchive-commons:1.1.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.openwayback:openwayback-cdx-server:2.0.0.BETA.2
      +-org.netpreserve.commons:webarchive-commons:1.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.commons:webarchive-commons:1.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.openwayback:openwayback-access-control-core:1.0.2
      +-org.netpreserve.commons:webarchive-commons:1.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.commons:webarchive-commons:1.1.4

[WARNING] 
Dependency convergence error for org.apache.commons:commons-math3:3.1.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.commons:commons-math3:3.1.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.commons:commons-math3:3.4.1

[WARNING] 
Dependency convergence error for io.netty:netty-all:4.0.23.Final paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-io.netty:netty-all:4.0.23.Final
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.spark:spark-network-common_2.10:1.6.0-cdh5.7.1
      +-io.netty:netty-all:4.0.29.Final
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.netty:netty-all:4.0.29.Final

[WARNING] 
Dependency convergence error for org.slf4j:slf4j-api:1.6.4 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro:1.7.6-cdh5.7.1
        +-org.slf4j:slf4j-api:1.6.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro-ipc:1.7.6-cdh5.7.1
        +-org.slf4j:slf4j-api:1.6.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro-ipc:1.7.6-cdh5.7.1
        +-org.slf4j:slf4j-api:1.6.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.slf4j:slf4j-api:1.6.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
            +-org.apache.directory.server:apacheds-i18n:2.0.0-M15
              +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
            +-org.apache.directory.api:api-asn1-api:1.0.0-M20
              +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
            +-org.apache.directory.api:api-util:1.0.0-M20
              +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
            +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.curator:curator-client:2.7.1
          +-org.slf4j:slf4j-api:1.7.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-shuffle:2.6.0-cdh5.7.1
          +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-jobclient:2.6.0-cdh5.7.1
        +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.slf4j:jul-to-slf4j:1.7.5
      +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.slf4j:jcl-over-slf4j:1.7.5
      +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.slf4j:slf4j-log4j12:1.7.5
      +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-slf4j_2.10:2.2.3-shaded-protobuf
      +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-core:3.1.2
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-jvm:3.1.2
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-json:3.1.2
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-graphite:3.1.2
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-hdfs:0.8.2
        +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-s3:0.8.2
        +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-local:0.8.2
        +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.intel.chimera:chimera:0.9.2
      +-org.slf4j:slf4j-api:1.7.10
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:netcdf4:4.5.5
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:grib:4.5.5
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-org.quartz-scheduler:quartz:2.2.0
        +-org.slf4j:slf4j-api:1.6.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-net.sf.ehcache:ehcache-core:2.6.2
        +-org.slf4j:slf4j-api:1.6.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:httpservices:4.5.5
      +-org.slf4j:slf4j-api:1.7.7

[WARNING] 
Dependency convergence error for joda-time:joda-time:2.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-joda-time:joda-time:2.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-de.jollyday:jollyday:0.4.7
      +-joda-time:joda-time:2.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-edu.ucar:udunits:4.5.5
        +-joda-time:joda-time:2.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-joda-time:joda-time:2.2

[WARNING] 
Dependency convergence error for com.fasterxml.jackson.core:jackson-databind:2.2.3 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-aws:2.6.0-cdh5.7.1
        +-com.fasterxml.jackson.core:jackson-databind:2.2.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-aws:2.6.0-cdh5.7.1
        +-com.amazonaws:aws-java-sdk-s3:1.10.6
          +-com.amazonaws:aws-java-sdk-core:1.10.6
            +-com.fasterxml.jackson.core:jackson-databind:2.5.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.spark:spark-network-shuffle_2.10:1.6.0-cdh5.7.1
      +-com.fasterxml.jackson.core:jackson-databind:2.2.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-com.fasterxml.jackson.core:jackson-databind:2.3.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-json:3.1.2
      +-com.fasterxml.jackson.core:jackson-databind:2.4.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.core:jackson-databind:2.2.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.module:jackson-module-scala_2.10:2.2.3
      +-com.fasterxml.jackson.core:jackson-databind:2.2.3

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message:
Failed while enforcing releasability the error(s) are [
Dependency convergence error for com.google.guava:guava:11.0.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.curator:curator-client:2.7.1
          +-com.google.guava:guava:16.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.apache.hadoop:hadoop-yarn-client:2.6.0-cdh5.7.1
            +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.apache.hadoop:hadoop-yarn-server-common:2.6.0-cdh5.7.1
            +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-yarn-api:2.6.0-cdh5.7.1
        +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.curator:curator-recipes:2.7.1
      +-org.apache.curator:curator-framework:2.7.1
        +-com.google.guava:guava:16.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.curator:curator-recipes:2.7.1
      +-com.google.guava:guava:16.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-hdfs:0.8.2
        +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-s3:0.8.2
        +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-local:0.8.2
        +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.intel.chimera:chimera:0.9.2
      +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-com.google.guava:guava:14.0.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-com.google.guava:guava:17.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-com.google.guava:guava:11.0.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-tl.lin:lintools-datatypes:1.0.0
    +-com.google.guava:guava:18.0
, 
Dependency convergence error for javax.xml.bind:jaxb-api:2.2.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-javax.xml.bind:jaxb-api:2.2.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-de.jollyday:jollyday:0.4.7
      +-javax.xml.bind:jaxb-api:2.2.7
, 
Dependency convergence error for commons-lang:commons-lang:2.6 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.apache.hadoop:hadoop-yarn-client:2.6.0-cdh5.7.1
            +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-yarn-api:2.6.0-cdh5.7.1
        +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-commons-lang:commons-lang:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-commons-lang:commons-lang:2.4
, 
Dependency convergence error for org.apache.httpcomponents:httpcore:4.1.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-net.java.dev.jets3t:jets3t:0.9.0
      +-org.apache.httpcomponents:httpcore:4.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-org.apache.httpcomponents:httpcore:4.2.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:httpservices:4.5.5
      +-org.apache.httpcomponents:httpcore:4.2.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:httpservices:4.5.5
      +-org.apache.httpcomponents:httpmime:4.2.6
        +-org.apache.httpcomponents:httpcore:4.2.5
, 
Dependency convergence error for org.xerial.snappy:snappy-java:1.0.4.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro:1.7.6-cdh5.7.1
        +-org.xerial.snappy:snappy-java:1.0.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.xerial.snappy:snappy-java:1.0.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.xerial.snappy:snappy-java:1.0.5
, 
Dependency convergence error for xml-apis:xml-apis:1.3.03 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-com.io7m.xom:xom:1.2.10
      +-xml-apis:xml-apis:1.3.03
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-com.io7m.xom:xom:1.2.10
      +-xalan:xalan:2.7.0
        +-xml-apis:xml-apis:1.0.b2
, 
Dependency convergence error for org.apache.commons:commons-lang3:3.3.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.commons:commons-lang3:3.3.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.apache.commons:commons-lang3:3.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-hdfs:0.8.2
        +-org.apache.commons:commons-lang3:3.0
, 
Dependency convergence error for com.google.code.gson:gson:2.2.4 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-com.google.code.gson:gson:2.2.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-tl.lin:lintools-datatypes:1.0.0
    +-com.google.code.gson:gson:2.3.1
, 
Dependency convergence error for org.apache.tika:tika-core:1.9 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-core:1.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.tika:tika-core:1.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.gagravarr:vorbis-java-tika:0.6
      +-org.apache.tika:tika-core:1.5
, 
Dependency convergence error for io.netty:netty:3.6.2.Final paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-io.netty:netty:3.6.2.Final
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf
      +-io.netty:netty:3.6.6.Final
, 
Dependency convergence error for commons-codec:commons-codec:1.4 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-net.java.dev.jets3t:jets3t:0.9.0
      +-commons-codec:commons-codec:1.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-commons-codec:commons-codec:1.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.poi:poi:3.12
      +-commons-codec:commons-codec:1.9
, 
Dependency convergence error for com.thoughtworks.paranamer:paranamer:2.3 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro:1.7.6-cdh5.7.1
        +-com.thoughtworks.paranamer:paranamer:2.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-com.thoughtworks.paranamer:paranamer:2.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.module:jackson-module-scala_2.10:2.2.3
      +-com.thoughtworks.paranamer:paranamer:2.3
, 
Dependency convergence error for commons-logging:commons-logging:1.1.3 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.intel.chimera:chimera:0.9.2
      +-commons-logging:commons-logging:1.1.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.springframework:spring-core:3.0.6.RELEASE
      +-commons-logging:commons-logging:1.1.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.pdfbox:pdfbox:1.8.9
      +-org.apache.pdfbox:fontbox:1.8.9
        +-commons-logging:commons-logging:1.1.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.pdfbox:pdfbox:1.8.9
      +-commons-logging:commons-logging:1.1.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-com.github.junrar:junrar:0.7
      +-org.apache.commons:commons-vfs2:2.0
        +-commons-logging:commons-logging:1.1.1
, 
Dependency convergence error for commons-net:commons-net:3.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-commons-net:commons-net:3.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-commons-net:commons-net:2.2
, 
Dependency convergence error for org.scala-lang:scala-library:2.10.5 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.scalatest:scalatest_2.10:2.2.5
    +-org.scala-lang:scala-library:2.10.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.scala-lang:scala-library:2.10.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-com.chuusai:shapeless_2.10.5:2.0.0
    +-org.scala-lang:scala-library:2.10.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.twitter:chill_2.10:0.5.0
      +-org.scala-lang:scala-library:2.10.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf
      +-org.spark-project.akka:akka-actor_2.10:2.2.3-shaded-protobuf
        +-org.scala-lang:scala-library:2.10.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf
      +-org.scala-lang:scala-library:2.10.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-slf4j_2.10:2.2.3-shaded-protobuf
      +-org.scala-lang:scala-library:2.10.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.scala-lang:scala-library:2.10.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-org.json4s:json4s-ast_2.10:3.2.10
          +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-org.scala-lang:scalap:2.10.0
          +-org.scala-lang:scala-compiler:2.10.0
            +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-org.json4s:json4s-core_2.10:3.2.10
        +-org.scala-lang:scalap:2.10.0
          +-org.scala-lang:scala-compiler:2.10.0
            +-org.scala-lang:scala-reflect:2.10.5
              +-org.scala-lang:scala-library:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.module:jackson-module-scala_2.10:2.2.3
      +-org.scala-lang:scala-library:2.10.2
, 
Dependency convergence error for org.apache.zookeeper:zookeeper:3.4.5-cdh5.7.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.zookeeper:zookeeper:3.4.5-cdh5.7.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.curator:curator-client:2.7.1
          +-org.apache.zookeeper:zookeeper:3.4.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.zookeeper:zookeeper:3.4.5-cdh5.7.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.apache.hadoop:hadoop-yarn-server-common:2.6.0-cdh5.7.1
            +-org.apache.zookeeper:zookeeper:3.4.5-cdh5.7.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.curator:curator-recipes:2.7.1
      +-org.apache.curator:curator-framework:2.7.1
        +-org.apache.zookeeper:zookeeper:3.4.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.curator:curator-recipes:2.7.1
      +-org.apache.zookeeper:zookeeper:3.4.6
, 
Dependency convergence error for xerces:xercesImpl:2.9.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-xerces:xercesImpl:2.9.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-com.io7m.xom:xom:1.2.10
      +-xerces:xercesImpl:2.8.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-com.syncthemall:boilerpipe:1.2.2
    +-net.sourceforge.nekohtml:nekohtml:1.9.20
      +-xerces:xercesImpl:2.10.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-com.syncthemall:boilerpipe:1.2.2
    +-xerces:xercesImpl:2.11.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-xerces:xercesImpl:2.11.0
, 
Dependency convergence error for com.google.code.findbugs:jsr305:3.0.0 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-com.google.code.findbugs:jsr305:3.0.0
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.spark:spark-network-common_2.10:1.6.0-cdh5.7.1
      +-com.google.code.findbugs:jsr305:1.3.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.spark:spark-unsafe_2.10:1.6.0-cdh5.7.1
      +-com.google.code.findbugs:jsr305:1.3.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.google.code.findbugs:jsr305:1.3.9
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.module:jackson-module-scala_2.10:2.2.3
      +-com.google.code.findbugs:jsr305:2.0.1
, 
Dependency convergence error for org.apache.commons:commons-compress:1.4.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro:1.7.6-cdh5.7.1
        +-org.apache.commons:commons-compress:1.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.commons:commons-compress:1.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-org.apache.commons:commons-compress:1.4.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-org.apache.commons:commons-compress:1.9
, 
Dependency convergence error for junit:junit:4.12 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-junit:junit:4.12
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.openwayback:openwayback-access-control-core:1.0.2
      +-junit:junit:4.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.openwayback:openwayback-access-control-core:1.0.2
      +-org.codehaus.jettison:jettison:1.0-beta-1
        +-junit:junit:3.8.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-com.googlecode.json-simple:json-simple:1.1.1
      +-junit:junit:4.10
, 
Dependency convergence error for org.apache.httpcomponents:httpclient:4.2.5 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.httpcomponents:httpclient:4.2.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-aws:2.6.0-cdh5.7.1
        +-com.amazonaws:aws-java-sdk-s3:1.10.6
          +-com.amazonaws:aws-java-sdk-core:1.10.6
            +-org.apache.httpcomponents:httpclient:4.3.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-net.java.dev.jets3t:jets3t:0.9.0
      +-org.apache.httpcomponents:httpclient:4.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:httpservices:4.5.5
      +-org.apache.httpcomponents:httpclient:4.2.6
, 
Dependency convergence error for org.jsoup:jsoup:1.7.3 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.jsoup:jsoup:1.7.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:grib:4.5.5
      +-org.jsoup:jsoup:1.7.2
, 
Dependency convergence error for org.netpreserve.commons:webarchive-commons:1.1.2 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.openwayback:openwayback-cdx-server:2.0.0.BETA.2
      +-org.netpreserve.commons:webarchive-commons:1.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.commons:webarchive-commons:1.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.openwayback:openwayback-core:2.0.0.BETA.2
    +-org.netpreserve.openwayback:openwayback-access-control-core:1.0.2
      +-org.netpreserve.commons:webarchive-commons:1.1.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.netpreserve.commons:webarchive-commons:1.1.4
, 
Dependency convergence error for org.apache.commons:commons-math3:3.1.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.commons:commons-math3:3.1.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.commons:commons-math3:3.4.1
, 
Dependency convergence error for io.netty:netty-all:4.0.23.Final paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-hdfs:2.6.0-cdh5.7.1
        +-io.netty:netty-all:4.0.23.Final
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.spark:spark-network-common_2.10:1.6.0-cdh5.7.1
      +-io.netty:netty-all:4.0.29.Final
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.netty:netty-all:4.0.29.Final
, 
Dependency convergence error for org.slf4j:slf4j-api:1.6.4 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro:1.7.6-cdh5.7.1
        +-org.slf4j:slf4j-api:1.6.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro-ipc:1.7.6-cdh5.7.1
        +-org.slf4j:slf4j-api:1.6.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.apache.avro:avro-ipc:1.7.6-cdh5.7.1
        +-org.slf4j:slf4j-api:1.6.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.avro:avro-mapred:1.7.6-cdh5.7.1
      +-org.slf4j:slf4j-api:1.6.4
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
            +-org.apache.directory.server:apacheds-i18n:2.0.0-M15
              +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
            +-org.apache.directory.api:api-asn1-api:1.0.0-M20
              +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
            +-org.apache.directory.api:api-util:1.0.0-M20
              +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-auth:2.6.0-cdh5.7.1
          +-org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15
            +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-common:2.6.0-cdh5.7.1
        +-org.apache.curator:curator-client:2.7.1
          +-org.slf4j:slf4j-api:1.7.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-common:2.6.0-cdh5.7.1
          +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-mapreduce-client-shuffle:2.6.0-cdh5.7.1
          +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-app:2.6.0-cdh5.7.1
        +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.apache.hadoop:hadoop-yarn-common:2.6.0-cdh5.7.1
          +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-core:2.6.0-cdh5.7.1
        +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-mapreduce-client-jobclient:2.6.0-cdh5.7.1
        +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.slf4j:jul-to-slf4j:1.7.5
      +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.slf4j:jcl-over-slf4j:1.7.5
      +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.slf4j:slf4j-log4j12:1.7.5
      +-org.slf4j:slf4j-api:1.7.5
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.spark-project.akka:akka-slf4j_2.10:2.2.3-shaded-protobuf
      +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-core:3.1.2
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-jvm:3.1.2
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-json:3.1.2
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-graphite:3.1.2
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-hdfs:0.8.2
        +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-s3:0.8.2
        +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.tachyonproject:tachyon-client:0.8.2
      +-org.tachyonproject:tachyon-underfs-local:0.8.2
        +-org.slf4j:slf4j-api:1.7.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.intel.chimera:chimera:0.9.2
      +-org.slf4j:slf4j-api:1.7.10
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:netcdf4:4.5.5
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:grib:4.5.5
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-org.slf4j:slf4j-api:1.7.7
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-org.quartz-scheduler:quartz:2.2.0
        +-org.slf4j:slf4j-api:1.6.6
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-net.sf.ehcache:ehcache-core:2.6.2
        +-org.slf4j:slf4j-api:1.6.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:httpservices:4.5.5
      +-org.slf4j:slf4j-api:1.7.7
, 
Dependency convergence error for joda-time:joda-time:2.1 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-joda-time:joda-time:2.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-edu.stanford.nlp:stanford-corenlp:3.4.1
    +-de.jollyday:jollyday:0.4.7
      +-joda-time:joda-time:2.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-edu.ucar:udunits:4.5.5
        +-joda-time:joda-time:2.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.tika:tika-parsers:1.9
    +-edu.ucar:cdm:4.5.5
      +-joda-time:joda-time:2.2
, 
Dependency convergence error for com.fasterxml.jackson.core:jackson-databind:2.2.3 paths to dependency are:
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-aws:2.6.0-cdh5.7.1
        +-com.fasterxml.jackson.core:jackson-databind:2.2.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1
      +-org.apache.hadoop:hadoop-aws:2.6.0-cdh5.7.1
        +-com.amazonaws:aws-java-sdk-s3:1.10.6
          +-com.amazonaws:aws-java-sdk-core:1.10.6
            +-com.fasterxml.jackson.core:jackson-databind:2.5.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.apache.spark:spark-network-shuffle_2.10:1.6.0-cdh5.7.1
      +-com.fasterxml.jackson.core:jackson-databind:2.2.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-org.json4s:json4s-jackson_2.10:3.2.10
      +-com.fasterxml.jackson.core:jackson-databind:2.3.1
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-io.dropwizard.metrics:metrics-json:3.1.2
      +-com.fasterxml.jackson.core:jackson-databind:2.4.2
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.core:jackson-databind:2.2.3
and
+-io.archivesunleashed:aut:0.1.0-SNAPSHOT
  +-org.apache.spark:spark-core_2.10:1.6.0-cdh5.7.1
    +-com.fasterxml.jackson.module:jackson-module-scala_2.10:2.2.3
      +-com.fasterxml.jackson.core:jackson-databind:2.2.3

Create tests for ComputeImageSize.scala

ComputeImageSize.scala has no test coverage. We need to create tests for it.

Do we need pythonconverters/ArcRecordConverter.scala? If so, tests. If not, delete it.

There is no test coverage on ArcRecordConverter.scala.

Improve RemoveHttpHeader.scala test coverage

RemoveHttpHeader.scala test coverage can be improved.