In this project you will get all the data yourself! 😱 By now, getting pre-made databases is way too simple for you, it's high time you start leveraging your freshly learnt skills, and bring your inner hacker into practice. Discover all the possibilities that web data has to offer; use your creativity and discuss in group how you can reach the next level!
There will be plenty of projects where the focus will be on doing analysis. For this one, we want you to develop your own product or new features for an existing business. You are welcome to use the analytical tools you have learned as well, but the focus should be on functionality this time.
- Project Description
- Project Goals
- Requirements
- Deliverables
- Mentoring
- Schedule
- Json Structure
- Presentation
- Tips & Tricks
- Resources
In this project, you will choose a topic and collect all the data by yourself. Luckily there's hundreds of Free API's where you can pull data from, and for the cases where there's no API, you'll have a chance to practice your cool web-scraping skills. If needed, you can later enrich your data with other sources of data. Always with a business sense in mind, you'll have a chance to dive deep into a topic of your choosing and come up with a new product/ features. You don't need to reinvent the wheel, sometimes looking at things through a different perspective is all it takes to develop great solutions for specific users.
- Learn how to develop an interesting question and find the data to answer it.
- Learn how to obtain data from different sources, including APIs, web-scraping and maybe even building some automated scrapers.
- Your data and pipeline should serve as the core development of the project, enabling to build your business idea.
- In case you need additional data to improve the quality of your project, feel free to enrich with other open-source databases available online.
- You must plan your project. That is why creating a Kanban or Trello Board is mandatory. You have a template for Trello here.
- You CAN'T CODE until you project is planned.
- Create a .gitignore file and include it in your repository.
- Your project must include data from (at least) 2 different data sources (APIs & web, dataset & APIs, ...)
You are required to turn in the following:
- GitHub Repository with scripts and notebooks used, README and a requirements.txt file.
- README file should have a description of your project and how does the pipeline work.
- Requirements file should have all the packages used for someone to run your scripts/notebooks. Always consider that when someone uses your code, they'll do it in a new environment, where the only packages that should be installed are the ones your project requires to be ran. Ideally it should also be specified the package version (ex:
scipy==1.4.1) - Links to any external data you used (should also be on the README file) .
- Slides for your presentation.
Your TA's and LT will be your mentors! We will:
- Follow your project in general, will be the second person that knows more about the project, after your group.
- Check if you are following the tasks, your blockers, etc
- Help/support you in specific questions.
Your mentor is NOT meant to:
- Know everything.
- Be your manager. You have to be the responsible person to do the tasks!
This one is on you! You have already a couple of projects in your bag and are getting better and better at organising and working as a team, let's keep building that momentum.
The README will be your paper and exists to act as a guide of your project. From describing the idea to having a walk-through on how to use your scripts, the README is the welcome file for any user that encounters your project.
The structure should be:
- Title of the project
- Introduction to your project.
- Explanation of how your pipeline works (API's + web-scraping depending on each case).
- Links to sources of external data you used to enrich your project (if aplicable).
- Conclusions after your analysis/ product/ feature(s) development.
- Further questions.
You will have 10 minutes to present your project. The below are some ideas for slides you could include in your presentation; those marked with an (M) are mandatory!
- (M) Title of the project
- (M) Business idea.
- (M) Technical developments.
- Main challenges & strengths.
- Product / Feature showcasing.
- Main insights.
- Questions you couldn't answer.
- Something funny that happened during the project.
- Things you learned during this project.
- If you could start from scratch, what would you do differently?
- First, choose your topic and look for API's that can help you providing data.
- Before you start coding and integrate more data, propose some interesting questions you could answer with the data you have.
AnyAPI
Top 50 Most Popular APIs on RapidAPI
18 Fun APIs For Your Next Project
WeatherBit
Strava
GitHub
Twitter
LastFM
Spotify
NYTimes
News
Reddit
Medium
Twitch
IGDB
OMDB
GIPHY
StackExchange
YouTube
TheSportsDB
NBA API
Data Analysis with Python
The Best Mario Kart Character According To Data Science
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent