๐ฐ๐ท ํ๊ธ ๋ฌธ์๋ ์๋๋ฅผ ๋ด์ฃผ์ธ์!
This project contains the following containers:
-
postgres
- Image: postgres:13
- Database Port: 5432
- User: airflow
- PWD: airflow
- References: https://hub.docker.com/_/postgres
-
redis
- Image: redis:7.2-bookworm
-
airflow-webserver
- Image: apache/airflow:2.9.2
- Port: 8080
-
spark: Spark Master
- Image: bitnami/spark:3.1.2
- Port: 8181
- References:
-
spark-worker-*: Spark workers
- Image: bitnami/spark:3.1.2
- References:
-
jupyter-spark: Jupyter notebook with pyspark for interactive development.
- Image: jupyter/pyspark-notebook:spark-3.1.2
- Port: 8888
- References:
- https://hub.docker.com/layers/jupyter/pyspark-notebook/spark-3.1.2/images/sha256-37398efc9e51f868e0e1fde8e93df67bae0f9c77d3d3ce7fe3830faeb47afe4d?context=explore
- https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-pyspark-notebook
- https://hub.docker.com/r/jupyter/pyspark-notebook/tags/
- run in background with build
docker-compose up --build -d
- check websites Airflow: http://localhost:8080
user: airflow
password: airflow
Spark Master: http://localhost:8181
Postgres - Database airflow:
- Server: localhost:5432
- Database: airflow
- User: airflow
- Password: airflow
Jupyter Notebook: http://127.0.0.1:8888
Following command will show you the URLs of running servers with their tokens, which you can copy and paste into your browser.
docker logs -f docker-airflow-spark-jupyter-spark-1
Safely way to stop Spark container
docker stop $(docker-compose ps | grep spark | awk '{print $1}')
or using Docker Compose:
docker-compose ps | grep spark | awk '{print $1}'
From here, This document written in korean ๐
๋ณธ ํ๋ก์ ํธ๋ ๋ค์ ์ปจํ ์ด๋๋ค๋ก ๊ตฌ์ฑ๋์ด ์์ต๋๋ค :)
-
postgres
- Image: postgres:13
- Database Port: 5432
- User: airflow
- PWD: airflow
- References: https://hub.docker.com/_/postgres
-
redis
- Image: redis:7.2-bookworm
-
airflow-webserver
- Image: apache/airflow:2.9.2
- Port: 8080
-
spark: Spark Master
- Image: bitnami/spark:3.1.2
- Port: 8181
- References:
-
spark-worker-*: Spark workers
- Image: bitnami/spark:3.1.2
- References:
-
jupyter-spark: Jupyter notebook with pyspark for interactive development.
- Image: jupyter/pyspark-notebook:spark-3.1.2
- Port: 8888
- References:
- https://hub.docker.com/layers/jupyter/pyspark-notebook/spark-3.1.2/images/sha256-37398efc9e51f868e0e1fde8e93df67bae0f9c77d3d3ce7fe3830faeb47afe4d?context=explore
- https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-pyspark-notebook
- https://hub.docker.com/r/jupyter/pyspark-notebook/tags/
- Docker Demon (Docker Desktop ์ด ๋ก์ปฌ์ ์ค์น๋์ด ์์ด์ผ ํฉ๋๋ค.)
- ๋ฐฑ๊ทธ๋ผ์ด๋์์ ์คํํ๋ ค๋ฉด ๋ณธ ํ๋ก์ ํธ ์ต์๋จ ์์น์์ ๋ค์ ๋ช ๋ น์ด๋ฅผ ์คํ์์ผ์ฃผ์ธ์
docker-compose up -d
- ๊ฐ ์๋ฒ ์ ์ํด์ ํ์ธํด์ฃผ์๋ฉด ๋ฉ๋๋ค. ๊ฐ ์๋ฒ๊ฐ ์ฌ๋ผ์ค๋๋ฐ ์๊ฐ์ด ๊ฑธ๋ฆด ์ ์์ผ๋ Docker Desktop ์์ ๋ก๊ทธ ํ์ธ ํ๋ฉด์ ์คํํด์ฃผ์๋ฉด ๋์ฑ ์ข์ต๋๋ค.
Airflow: http://localhost:8080
user: airflow
password: airflow
Spark Master: http://localhost:8181
Postgres - Database Connection Test:
- Server: localhost:5432
- Database: airflow
- User: airflow
- Password: airflow
Jupyter Notebook: http://127.0.0.1:8888
์ฃผํผํฐ ๋ ธํธ๋ถ์ ์๋ฒ ๋ก๊ทธ๋ฅผ ํตํด ํ์ธ๋๋ ํ ํฐ์ ๋ณด๋ฅผ ๊ธฐ์ ํด์ฃผ์ธ์. ๋ค์ ๋ช ๋ น์ด๋ก ๋ก๊ทธ ํ์ธํด์ฃผ์๋ฉด ๋ฉ๋๋ค.
docker logs -f docker-airflow-spark-jupyter-spark-1
## ๋ก๊ทธ์์ ๋ค์๊ณผ ๊ฐ์ด ํ์ธ ๋๋ ์ ๋ณด ์ค์์ ex) http://localhost:8888/?token=c8de56fa...
## token ๊ฐ ์ ๋ณด๋ฅผ ์
๋ ฅํด์ฃผ์๋ฉด ๋ฉ๋๋ค.
๋์ปค ๋ช ๋ น์ด๋ก ์ข ๋ฃํ๊ธฐ
docker stop $(docker-compose ps | grep spark | awk '{print $1}')
์๋๋ฉด... Docker Compose ๋ช ๋ น์ด๋ก ์ข ๋ฃํ๊ธฐ
docker-compose ps | grep spark | awk '{print $1}'