tritondatacenter/hadoop-manta

Hadoop Filesystem Driver for Manta

Manta Filesystem for Hadoop

Introduction

This project provides a Hadoop FileSystem for the open source Manta object store. Unlike other object stores, Manta is strongly consistent and uses a hierarchical file system (like a Unix filesystem) to organize file storage whereas S3/Swift use a key/value system. This model aligns closely with the Hadoop Filesystem model and it leaves us with very few divergences from the default behavior.

Quirks / Divergences

• There is no HDFS functionality natively implemented. All files are accessed via network data transfer.
• Manta URIs take the form of manta:///user.name/stor/path/to/file
• Configuration can be done using Hadoop configuration parameters, Java system properties or environment variables.
• Append is not supported.
• Truncate is only supported for truncating a file to zero bytes.
• Checksums are performed using md5.
• Checksums for portions of large files are done remotely using Manta jobs.
• Disk space use statistics are based on usage reports which aren't updated instantaneously.
• Globally the default replication factor is 2.
• No assumptions about blocksize are made.
• setWriteChecksum() isn't supported yet.

Run Requirements

• Java 8
• Hadoop 2.8.0
• A running instance of the Manta object store (available on the public cloud)

Build Requirements

• Java 8
• Maven 3.0+

Configuration

You will need to have the public/private keys needed to access Manta on the machine in which Hadoop is running. It is often best to verify that these keys are setup correctly using the Node.js Manta CLI.

Configuration will be done using the Hadoop configuration files or environment variables. Refer to the table below for the available configuration options.

Configuration Parameters

Configuration used are the same as Java Manta Client SDK. Hadoop-specific property keys use the same keys as system properties within the Java Manta SDK.

Installation

Please refer to the following guides for installing the Manta Filesystem for Hadoop on different Hadoop ecosystem applications.

• Apache Hadoop
• Apache Drill
• Apache Scoop

License

The Manta Filesystem for Hadoop is licensed under the Apache 2.0 license.

hadoop-manta's Issues

Backward compatible with 2.7?

I wish to use this adapter with Snappydata database. Do you know if it is backwards compatible?

It states it uses hadoop 2.7.

https://snappydatainc.github.io/snappydata/configuring_cluster/configuring_cluster/#hdfs

Any suggestions?

Inconsistent operations on paths containing characters outside of range

Holla,

Using hadoop-manta through Kafka Connect HDFS discloses a strange behavior.

Paths that contain characters that are outside of the ones allowed by Manta are created without warning. The out-of-range characters get escaped.

Ex.: ~~/stor/project/topics/+tmp/topic_name/partition=0 -> ~~/stor/project/topics/%2Btmp/topic_name/partition%3D0 (which is okay, I guess)

The problem arises when kafka-connect-hdfs tries to read back from there: Manta returns a 404.

[1:statusLine=HTTP/1.1 404 Not Found]
[2:serverCode=DirectoryDoesNotExist]
[3:server_message=/<account>/stor/project/topics/+tmp/topic_name/partition=0 does not exist]

Looks like the conversion happens silently on the Manta side, on creation. (I might be wrong I did not dig yet.)

Should:

• hadoop-manta escape the path somewhere like here before even involving Manta
• Manta be responsible and either: prevent creation of such items OR convert consistently

Please let me know if this is a real issue or/and if I am getting something wrong.

Regards,
Marc

Add documentation for setting up the Hadoop FS connector with Apache Hive

Revised changelog uses 2016 year

typo.