Read the blog post A quick demo of Apache Beam with Docker.
Prerequisites: docker
and docker-compose
-
Clone this repo
-
Deploy cluster and see config/setup log output (best run in a screen session)
docker-compose up
or deploy as a daemon (and return)
docker-compose up -d
- Scale the cluster up or down to N TaskManagers
docker-compose scale taskmanager=<N>
Open the web UI on (exposed on port 48080)
open http://localhost:48080
or on Mac or Windows with docker-machine
open http://$(docker-machine ip default):48080
Next:
-
Click "Submit new Job" in the left menu
-
Flag the checkbox near
beam-starter-<ver>.jar
-
Click on "Submit" (or "Show Plan") -- no param is needed, but you can play with that too
- Access the JobManager node with SSH (exposed on port 220)
ssh root@$(docker-machine ip default) -p 220
The password is "secret"
-
Kill the cluster
docker-compose kill
-
Upload a jar to the cluster
scp -P 220 <your_jar> root@$(docker-machine ip default):/<your_path>
-
Run any Flink topology
ssh -p 220 root@$(docker-machine ip default) /usr/local/flink/bin/flink run -c <your_class> <your_jar> <your_params>
or ssh to the job manager and run the topology from there.
- The Web Dashboard is on port
48080
- The Web Client is on port
48081
- JobManager RPC port
6123
(default, not exposed to host) - TaskManagers RPC port
6121
(default, not exposed to host) - TaskManagers Data port
6122
(default, not exposed to host) - JobManager SSH
220
- TaskManagers SSH: randomly assigned port, check wih
docker ps
Edit the docker-compose.yml
file to edit port settings.
Clone this repo, then run sh build.sh
.
- Add the JAR to the front end during image building
- Forked from https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
- Apache Beam: https://beam.incubator.apache.org
- Apache Flink: https://flink.apache.org
- Starter repo for Apache Beam (included in this image): https://github.com/ecesena/beam-starter
- Blog: A quick demo of Apache Beam with Docker.