Question of framework selection

[chabane]

Question of framework selection

[kination]

@Chabane thanks! I'll keep look in to it.

[Chabane]

hello @djKooks

Good question, thanks

You are right, you can select one of them. The Spark project (Java or Scala) is the same as Flink project (Scala).

I will continue to develop the machine learning part in the upcoming months. I am in learning phase 👍

[kination]

@Chabane thanks for quick reply!
How can I select to run one of them? I couldn't find guide about it in README.

[Chabane]

there is an example here : https://github.com/Chabane/bigdata-playground#installation

run only the commands related to spark or flink and its dependencies.

to know which dependency you need to run, please refer to the diagram defined here https://github.com/Chabane/bigdata-playground#interactions--ongoing

[kination]

@Chabane okay. Is it mean that, if I want to run based on spark, will this be okay?

docker network create vnet
cd batch/spark && sbt clean package assembly && cd ../..
cd streaming/spark && sbt clean package assembly && cd ../..
cd docker
docker-compose -f mongo.yml -f zookeeper.yml -f kafka.yml -f hadoop-hbase.yml up -d
docker-compose -f dev/webapp.yml up -d
docker-compose -f dev/batch-spark.yml up -d
docker-compose -f dev/streaming-spark.yml up -d

Also, I got issue while following up above...

$ docker-compose -f dev/batch-spark.yml up -d
WARNING: Found orphan containers (webapp-dev) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Building batch-spark-dev
Step 1/12 : FROM openjdk:8-jre
8-jre: Pulling from library/openjdk
55cbf04beb70: Already exists
1607093a898c: Already exists
9a8ea045c926: Already exists
1290813abd9d: Pull complete
8a6b982ad6d7: Pull complete
abb029e68402: Pull complete
d068d0a738e5: Pull complete
42ee47bb0c52: Pull complete
Digest: sha256:16691ef8d07b275e0dbb3a49913ab02d2ede28b9fe01c13bb91af504a917a797
Status: Downloaded newer image for openjdk:8-jre
 ---> f38c4b9f5e8a
Step 2/12 : VOLUME /tmp
 ---> Running in 39747be9880b
Removing intermediate container 39747be9880b
 ---> 0116625f22e2
Step 3/12 : ENV SPARK_VERSION spark-2.3.0-bin-hadoop2.7
 ---> Running in 01bd521dc4c0
Removing intermediate container 01bd521dc4c0
 ---> 959e6ccf7164
Step 4/12 : ENV SPARK_HOME /usr/local/spark
 ---> Running in 7920565f9eb3
Removing intermediate container 7920565f9eb3
 ---> 0cfe31bf2294
Step 5/12 : ENV PARQUET_HOME $SPARK_HOME/parquet
 ---> Running in 58fd9fa73bff
Removing intermediate container 58fd9fa73bff
 ---> 275bd6c99419
Step 6/12 : RUN curl http://apache.mediamirrors.org/spark/spark-2.3.0/$SPARK_VERSION.tgz -o $SPARK_VERSION.tgz;          tar xzf $SPARK_VERSION.tgz -C /usr/local/;
 ---> Running in 5cc70023355a
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   334  100   334    0     0    313      0  0:00:01  0:00:01 --:--:--   313

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
ERROR: Service 'batch-spark-dev' failed to build: The command '/bin/sh -c curl http://apache.mediamirrors.org/spark/spark-2.3.0/$SPARK_VERSION.tgz -o $SPARK_VERSION.tgz;          tar xzf $SPARK_VERSION.tgz -C /usr/local/;' returned a non-zero code: 2
...

Could you find something?

[Chabane]

Nice one, the url is a dead link.

I updated the dockerfile. Should work now.

[kination]

@Chabane thanks. It seems work well now.
I just success to load airport UI, but couldn't see additional action or page after I press search button.
Do you have some kind of plan for this, or is it something wrong?

Actually, twitter account is not setup yet...(I'm waiting for acceptance). Can this effect it?

[Chabane]

There is no additional action now (except the streaming part).
The main purpose of this playground is to display the tweets that match the search flight using a machine learning app (stand by).