Explore storing continuous ingest bulk import files in S3

[apache]

Explore storing continuous ingest bulk import files in S3

[keith-turner]

I suspect the procedure for this would be the following :

• Generate data
  • Run lots of map reduce jobs to create multiple dirs for bulk import
  • Save split points to a file
  • distcp bulk import dirs and splits points from HDFS to S3 bucket
• Use data
  • distcp bulk import dirs from S3 to HDFS
  • get splits file from S3
  • split table with single tablet using splits file
  • import data

https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html#S3A

[keith-turner]

Current bulk import test docs : bulk-test.md

[keith-turner]

Below are some notes from copying bulk data from HDFS to S3

# bulk files were generated intp /tmp/bt dir in hdfs

# prep directory before distp, assuming all splits files are the same just keep one
hadoop fs -mv /tmp/bt/1/splits.txt /tmp/bt
hadoop fs -rm /tmp/bt/*/splits.txt
hadoop fs -rm /tmp/bt/*/files/_SUCCESS

# get the S3 libs on the local hadoop classpath
# edit following file and set : export HADOOP_OPTIONAL_TOOLS="hadoop-aws"
vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh 

# The remote map reduce jobs will need the S3 jars on the classpath, define the following for this.. may need to change for your version of hadoop
export LIBJARS=$HADOOP_HOME/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar,$HADOOP_HOME/share/hadoop/tools/lib/hadoop-aws-3.2.0.jar

# the following command will distcp files to bucket
hadoop distcp -libjars ${LIBJARS} -Dfs.s3a.access.key=$AWS_KEY -Dfs.s3a.secret.key=$AWS_SECRET hdfs://leader1:8020/tmp/bt s3a://$AWS_BUCKET/continuous-1000