hiroki11x/sparkmesossandbox

Practice repository like Apache Spark, Apache Mesos and Spring Boot

About

Hi! I'm Hiroki Naganuma, a Ph.D. candidate in Computer Science at Université de Montréal and Mila - Quebec AI Institute advised by Professor Ioannis Mitliagkas. My research interests center around large-scale parallelization in machine learning and understanding the training dynamics of deep neural networks.

Previously, I received B.Sc. from Tokyo Institute of Technology in 2017 and M.Sc. from Tokyo Institute of Technology in 2019. In previous years, I have been honored to closely work with Professor Rio Yokota as well as many other exceptional collaborators.

My CV can be found here (Last updated: Feb 2024.

Stats

sparkmesossandbox's Issues

Dynamodbが無反応

Macbook Airのlocal環境だったDynamodbを
MacbookProにcloneして使おうとしたらダメだった

Task not serializable

Sandbox01
にてRDDとしてtextfileを読みDynamodbに書き込もうとすると怒られた

txt->DynamodbTableならSpark使わなくていいかも

http://stackoverflow.com/questions/22592811/task-not-serializable-java-io-notserializableexception-when-calling-function-ou

java.lang.NoSuchMethodError: com.amazonaws.ClientConfiguration.withRetryPolicy

AmazonDynamoDBClient client = new AmazonDynamoDBClient(new BasicAWSCredentials("AWS_ACCESS_KEY_ID", "yourSecretAccessKey"));
client.setEndpoint("http://localhost:8000");

みたいにしようとしても

Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.ClientConfiguration.withRetryPolicy(Lcom/amazonaws/retry/RetryPolicy;)Lcom/amazonaws/ClientConfiguration;
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientConfigurationFactory.getDefaultConfig(AmazonDynamoDBClientConfigurationFactory.java:31)
at com.amazonaws.ClientConfigurationFactory.getConfig(ClientConfigurationFactory.java:36)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.(AmazonDynamoDBClient.java:311)
at dynamo.DynamodbSample01$DynamoUtils.setupDynamoClientConnection(DynamodbSample01.java:72)
at dynamo.DynamodbSample01.main(DynamodbSample01.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

って言われる

SparkのRDDをDynamodbから生成

Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;

awsのsdkのバージョンを最新にすると

Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;

と言われる、、

Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError

http://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/developerguide/DynamoDBLocal.html
から取得した最新版だと　

$ java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -port 8001
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

みたいになる

RDDからDynamodbの要素にアクセス

Exception in thread "main" com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One of the required keys was not given a value (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException

#4

は解決したものの、今度は

Exception in thread "main" com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One of the required keys was not given a value (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: hogehoge)

と言われた

static public void registerTable(DynamoDB dynamoDB)内の

UpdateItemOutcome outcome = table.updateItemで何か言われてる(サンプルのscalaをjavaに変換してコピペしたままなのでよくわかってない)