microsoft/masc

Microsoft's contributions for Spark with Apache Accumulo

Microsoft MASC, an Apache Spark connector for Apache Accumulo

The goal of this repository is to facilitate the use of Apache Spark and its machine learning ecosystem with Apache Accumulo as an external data source.

Contents

• The connector provides connectivity to read from / write to Accumulo using Spark. See the README for more details about supported functionality.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Build

License

All code provided, except where otherwise documented in OpenSource and NOTICE, is covered by the Apache License 2.0

Trademarks

Apache®, Apache Spark, Apache Accumulo and Accumulo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

masc's Issues

Evaluate performance improvement by using SupportsWriteInternalRow

From the api docs:

A mix-in interface for DataSourceWriter. Data source writers can implement this interface to write InternalRow directly and avoid the row conversion at Spark side. This is an experimental and unstable interface, as InternalRow is not public and may get changed in the future Spark versions.

Show examples of setting table split-points

In the example notebook, use an example of making pre-split of the table, e.g. properties['splits'] = "1,3,5,7"

Dependency convergence error

Add dependency convergence enforcer and resolve issues

Document to specify the maven version requirement

(On CentOS)
sudo yum install maven installs maven version 3.0.5 and I got this error while building a jar:

[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.2.0:compile (default) on project accumulo-spark-datasource: The plugin net.alchim31.maven:scala-maven-plugin:4.2.0 requires Maven version 3.3.9 -> [Help 1]

I followed this link to install mvn 3.3.9 and now could build without errors.

Column pruning is broken

The current maintains a single array for all the cells collected into a row.
This needs to be extended as each stage (filter, mleap, output) might need access to more fields than ultimately exported (e.g. filter: x> 0, select: y)

Use MLeap Avro to simplify MLeap/Accumulo integration

https://mvnrepository.com/artifact/ml.combust.mleap/mleap-avro (DefaultRowReader/DefaultRowWriter)

Id in schema when reading from Accumulo

If "id" is part of the schema, then spark.read.format("com.microsoft.accumulo").options(props.asScala).schema(schema).load fails with an error.

Explore adding root pom

It may be useful to users to have a root pom as a main entry point without having to read any docs or explore where the code is located.

Example notebook read / write format package path should be updated

The example still uses org.apache.accumulo package names and should be updated to use the new one which is com.microsoft.accumulo

Support .mode when writing

Support .mode() when writing. E.g.

df.write.mode("overwrite")...

Test error

mvn test gets following error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (spark-shade-jar) on project accumulo-spark-datasource: Error creating shaded jar: /data/home/jumin/git/masc/connector/zipfs/target/classes (Is a directory) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :accumulo-spark-datasource