Clean Marine

This web app serves as a centralized hub for information about local beach clean up events, originally created for the HOTHX hackathon, hosted by UCLA ACM Hack in 2023.

Clean Marine was built using the Python Django full-stack webframework for hosting th SQLite database and web server, webscraping scripts using Selenium and Beautiful Soup, and the Google Maps API to provide users a visual guide on locating nearby events. Additionally, this app has been Dockerizied allowing it to be run in any environment (with Docker installed) with a simplified setup process. This feature was implemented for Paramount Data and Decision Science Group's Hackweek 2023 to research how to Dockerize multi-container applications.

HothX Presentation and Demo: https://youtu.be/zuwXPLttlSM

Docker Setup
Invoke Commands
- invoke scrape
- invoke dump
Database Manipulation
Conda Setup (Deprecated)
- Steps
- Updating dependencies
Resources

Docker Setup

Steps

Install Docker Desktop. Open Docker and restart the your terminal window.

Clone the repository and cd into it.

git clone https://github.com/ajtadeo/clean-marine.git
cd clean-marine

In the project root directory, create .env with the following credentials:
```
GOOGLE_MAPS_KEY=secret
SECRET_KEY=secret
```
Execute docker compose up in the project root directory.
Open the app at 0.0.0.0:8000
To stop the app, either click Ctrl-C in the terminal or run docker compose down in a seperate terminal.

Docker-ization Steps

Originally, Clean Marine was setup using a conda virtual environment. The code for this implementation can be viewed here. The following steps were taken to Dockerize the application to the current implementation:

Convert base.yml to requirements.txt.
```
pip freeze > requirements.txt
```
Fill out .env with the following credentials. If .env hasn't already been created, create it.
```
GOOGLE_MAPS_KEY=secret
SECRET_KEY=secret
```
NOTE: We can't use Docker Secrets because this app isn't configured as a Docker Swarm.

In the root directory, initialize Docker files and create an image of the application

docker init
    ? What application platform does your project use? Python
    ? What version of Python do you want to use? 3.11.4
    ? What port do you want your app to listen on? 8000
    ? What is the command to run your app? python manage.py runserver 0.0.0.0:8000

Now, the file structure should look like this:

clean-marine/           <-- Django project root     
├── clean-marine/       <-- clean-marine app, contains settings and configuration
├── dashboard/          <-- dashboard app
│   ├── static/
│   ├── templates/
│   ├── base.yml
│   ├── db.sqlite3
│   ├── dump.json
│   ├── manage.py
│   ├── notes.txt
│   └── tasks.py
├── components/         <-- components
│   └── event/
├── .gitignore
├── .dockerignore       <-- NEW
├── .env
├── .gitignore
├── compose.yaml        <-- NEW
├── Dockerfile          <-- NEW
├── manage.py
├── tasks.py
├── requirements.txt
└── README.md

Update compose.yaml to reference .env as the environment variables file.
```
services:
 django:
   build:
     context: .
   ports:
     - 8000:8000
   env_file:
     - ./.env
```
NOTE: Selenium must be added as a service to fix the broken webscrapers. This will be implemented in a future update.
In clean-marine/settings.py, add 0.0.0.0 to ALLOWED-HOSTS to let the user connect to the Django app through the isolated Docker environment.
```
# ...
ALLOWED_HOSTS = ['0.0.0.0']
# ...
```
Create the Docker image and run the app.
```
docker compose up
```
Visit your newly Docker-ized app at 0.0.0.0:8000 🥳!

NOTE: The Docker images in this multi-container app have the format clean-marine-[service].
To stop the app, either click Ctrl-C in the terminal or run docker compose down in a seperate terminal.

Making Changes

If changes are made to the Django app or the Docker configuration files, its necessary to remove the containers and images and create new ones.

Stop any running containers by either clicking Ctrl-C in the terminal or run docker compose down in a seperate terminal. These commands will also remove the containers.
Verify the containers were removed.
```
docker ps -a
```
Remove the image.
```
docker image rm clean-marine-django
```
Verify the image was removed.
```
docker images
```
Recreate the image.
```
docker compose up
```

Invoke Commands

Invoke commands are this app's custom CLI, used for simplifying tasks that are done regularly. Run these commands from the root directory clean-marine.

`invoke scrape`

Runs all web scrapers and updates the database with new information.
Currently implemented web scrapers:
- Surfrider Foundation

`invoke dump`

Dumps data from the SQLite database into dump.json in the project root directory.

Database Manipulation

View Table Structure

In the terminal:
```
python3 manage.py dbshell
```

In the dbshell:

.header on
.mode column
pragma table_info('dashboard_events');

Update Table Structure

Make changes to dashboard/models.py

In the terminal in the root directory:

python manage.py makemigrations
python manage.py migrate

Delete the Table

Sometimes it's best to just start from scratch and rerun the webscraper ;-;

In the terminal:
```
python manage.py dbshell
```
In the dbshell:
```
DROP TABLE dashboard_events;
```

Create the Table

In the terminal:
```
python manage.py dbshell
```

In the dbshell:

CREATE TABLE dashboard_events (
    id INTEGER PRIMARY KEY NOT NULL,
    eventname TEXT NOT NULL UNIQUE,
    organization TEXT NOT NULL,
    link TEXT NOT NULL,
    datetime TEXT,
    location TEXT,
    lng FLOAT,
    lat FLOAT
);

Conda Setup (Deprecated)

The code for this deprecated setup method is intended to be used at this release or earlier. Use the Docker setup for current implementation.

Steps

Install python3 and the Chrome driver for web scraping on your machine

Clone the repository and cd into it.

git clone https://github.com/ajtadeo/clean-marine.git
cd clean-marine

Set up the virtual environment.
- Create the virutal environment.
```
conda env create -f base.yml
```
- Verify that the virtual environment was created correctly.
```
conda info --envs
```
- Activate the virtual environment.
```
conda activate cm_base
```
Migrate and update the SQLite database. Currently, running ./manage.py makemigrations makes migrations and scrapes the web since there is no automated scraping capability.
```
./manage.py makemigrations
./manage.py migrate
```
Start the development server.
```
./manage.py runserver
```

Updating dependencies

Update dependencies under pip in base.yml.
Deactivate the virtual environment if it is currently running.
```
conda deactivate
```
Update the virtual environment.
- If you are removing a dependency, removing and recreating the virtual environment is necessary since --prune does not work as of August 9 2023.
```
conda remove --name cm_base --all
conda env create -f base.yml
```
- Otherwise, update the virtual environment as normal.
```
conda env update -f base.yml
```
Verify that the virtual environment was updated correctly.
```
conda info --envs
```
Activate the virtual environment.
```
conda activate cm_base
```
Verify that the new dependency was added to the virtual environment.
```
pip list --local
```

ajtadeo / clean-marine Goto Github PK

clean-marine's Introduction

Clean Marine

Table of Contents

Docker Setup

Steps

Docker-ization Steps

Making Changes

Invoke Commands

invoke scrape

invoke dump

Database Manipulation

View Table Structure

Update Table Structure

Delete the Table

Create the Table

Conda Setup (Deprecated)

Steps

Updating dependencies

Resources

clean-marine's People

Contributors

clean-marine's Issues

Recommend Projects

Recommend Topics

Recommend Org

`invoke scrape`

`invoke dump`