A journey into data engineering. This repository contains the code and notes for Python workflows such as API calls and web scraping. The results can be used to build an automated data pipeline in the cloud.
Tech used:
-
Pandas
pip install pandas
-
sqlalchemy
pip install sqlalchemy
-
beautifulsoup4
pip install beautifulsoup4
-
os
pip install os
-
requests
pip install requests
Datasets & sources:
-
cities_info ( www.wikipedia.org )
-
arrivals_info ( https://aerodatabox.p.rapidapi.com )
-
weather_info ( http://api.openweathermap.org )