This example of ETL pipeline, it is trying to store data that extracting in front-end into MySQL Database.
-
Python3
-
Lib: BeautifulSoup, urllib, pandas, pyyml
pip3 install requirements.txt
-
Install virtual environment
python3 -m venv demo
- Database: Mysql
- Install Mysql, MysqlConnector or Using Docker Container
docker pull mysql
docker run --name mysql_container -e MYSQL_ROOT_PASSWORD="RootAccount123" -d mysql
docker ps
- Database Management: phpMyadmin / Dbeaver
- Install phpmyadmin package for further manage databases
- Use default local IP and root user to login.
- Environment: Any
- Editor: Any
- Extract category in the menu, an output is list of url (Subcategory)
- Using each input as parameter for page_parsing, which is parsing data of every products on fixed number page (customable)
- Store raw data into MySQL
- Start the Python venv
(demo)
source demo/bin/activate
- Start the
main_build.py
file
python3 main_build.py
- clean raw data for useable
- create pipeline for scheduling run
- apply model, analysis