This web app serves as a centralized hub for information about local beach clean up events, originally created for the HOTHX hackathon, hosted by UCLA ACM Hack in 2023.
Clean Marine was built using the Python Django full-stack webframework for hosting th SQLite database and web server, webscraping scripts using Selenium and Beautiful Soup, and the Google Maps API to provide users a visual guide on locating nearby events. Additionally, this app has been Dockerizied allowing it to be run in any environment (with Docker installed) with a simplified setup process. This feature was implemented for Paramount Data and Decision Science Group's Hackweek 2023 to research how to Dockerize multi-container applications.
HothX Presentation and Demo: https://youtu.be/zuwXPLttlSM
- Install Docker Desktop. Open Docker and restart the your terminal window.
- Clone the repository and cd into it.
git clone https://github.com/ajtadeo/clean-marine.git cd clean-marine
- In the project root directory, create
.env
with the following credentials:GOOGLE_MAPS_KEY=secret SECRET_KEY=secret
- Execute
docker compose up
in the project root directory. - Open the app at
0.0.0.0:8000
- To stop the app, either click
Ctrl-C
in the terminal or rundocker compose down
in a seperate terminal.
Originally, Clean Marine was setup using a conda virtual environment. The code for this implementation can be viewed here. The following steps were taken to Dockerize the application to the current implementation:
-
Convert
base.yml
torequirements.txt
.pip freeze > requirements.txt
-
Fill out
.env
with the following credentials. If.env
hasn't already been created, create it.GOOGLE_MAPS_KEY=secret SECRET_KEY=secret
NOTE: We can't use Docker Secrets because this app isn't configured as a Docker Swarm.
-
In the root directory, initialize Docker files and create an image of the application
docker init ? What application platform does your project use? Python ? What version of Python do you want to use? 3.11.4 ? What port do you want your app to listen on? 8000 ? What is the command to run your app? python manage.py runserver 0.0.0.0:8000
Now, the file structure should look like this:
clean-marine/ <-- Django project root ├── clean-marine/ <-- clean-marine app, contains settings and configuration ├── dashboard/ <-- dashboard app │ ├── static/ │ ├── templates/ │ ├── base.yml │ ├── db.sqlite3 │ ├── dump.json │ ├── manage.py │ ├── notes.txt │ └── tasks.py ├── components/ <-- components │ └── event/ ├── .gitignore ├── .dockerignore <-- NEW ├── .env ├── .gitignore ├── compose.yaml <-- NEW ├── Dockerfile <-- NEW ├── manage.py ├── tasks.py ├── requirements.txt └── README.md
-
Update
compose.yaml
to reference.env
as the environment variables file.services: django: build: context: . ports: - 8000:8000 env_file: - ./.env
NOTE: Selenium must be added as a service to fix the broken webscrapers. This will be implemented in a future update.
-
In
clean-marine/settings.py
, add0.0.0.0
toALLOWED-HOSTS
to let the user connect to the Django app through the isolated Docker environment.# ... ALLOWED_HOSTS = ['0.0.0.0'] # ...
-
Create the Docker image and run the app.
docker compose up
Visit your newly Docker-ized app at
0.0.0.0:8000
🥳!NOTE: The Docker images in this multi-container app have the format
clean-marine-[service]
. -
To stop the app, either click
Ctrl-C
in the terminal or rundocker compose down
in a seperate terminal.
If changes are made to the Django app or the Docker configuration files, its necessary to remove the containers and images and create new ones.
-
Stop any running containers by either clicking
Ctrl-C
in the terminal or rundocker compose down
in a seperate terminal. These commands will also remove the containers. -
Verify the containers were removed.
docker ps -a
-
Remove the image.
docker image rm clean-marine-django
-
Verify the image was removed.
docker images
-
Recreate the image.
docker compose up
Invoke commands are this app's custom CLI, used for simplifying tasks that are done regularly. Run these commands from the root directory clean-marine
.
- Runs all web scrapers and updates the database with new information.
- Currently implemented web scrapers:
- Surfrider Foundation
- Dumps data from the SQLite database into
dump.json
in the project root directory.
- In the terminal:
python3 manage.py dbshell
- In the dbshell:
.header on .mode column pragma table_info('dashboard_events');
- Make changes to
dashboard/models.py
- In the terminal in the root directory:
python manage.py makemigrations python manage.py migrate
Sometimes it's best to just start from scratch and rerun the webscraper ;-;
- In the terminal:
python manage.py dbshell
- In the dbshell:
DROP TABLE dashboard_events;
- In the terminal:
python manage.py dbshell
- In the dbshell:
CREATE TABLE dashboard_events ( id INTEGER PRIMARY KEY NOT NULL, eventname TEXT NOT NULL UNIQUE, organization TEXT NOT NULL, link TEXT NOT NULL, datetime TEXT, location TEXT, lng FLOAT, lat FLOAT );
The code for this deprecated setup method is intended to be used at this release or earlier. Use the Docker setup for current implementation.
- Install python3 and the Chrome driver for web scraping on your machine
- Clone the repository and cd into it.
git clone https://github.com/ajtadeo/clean-marine.git cd clean-marine
- Set up the virtual environment.
- Create the virutal environment.
conda env create -f base.yml
- Verify that the virtual environment was created correctly.
conda info --envs
- Activate the virtual environment.
conda activate cm_base
- Create the virutal environment.
- Migrate and update the SQLite database. Currently, running
./manage.py makemigrations
makes migrations and scrapes the web since there is no automated scraping capability../manage.py makemigrations ./manage.py migrate
- Start the development server.
./manage.py runserver
- Update dependencies under
pip
inbase.yml
. - Deactivate the virtual environment if it is currently running.
conda deactivate
- Update the virtual environment.
- If you are removing a dependency, removing and recreating the virtual environment is necessary since
--prune
does not work as of August 9 2023.conda remove --name cm_base --all conda env create -f base.yml
- Otherwise, update the virtual environment as normal.
conda env update -f base.yml
- If you are removing a dependency, removing and recreating the virtual environment is necessary since
- Verify that the virtual environment was updated correctly.
conda info --envs
- Activate the virtual environment.
conda activate cm_base
- Verify that the new dependency was added to the virtual environment.
pip list --local