dse511-project3-team/dse511-project-3-code-repo

Git repository to manage the machine learning learning project-3 for DSE511.

dse511-project-3-code-repo's Issues

Create a branch titled 'Russ'

Create a branch for project 3 unique to group member Russ.

Data: Update Download Raw Data Script

We need to update the raw data generation script to directly download data from a web URL. This will help streamline the process.

EDA: Weather Condition

• Plot a chart showing the mean severity of accidents by varying weather conditions.

Impute weather related variables

Use KNN and OLS to impute missing weather variables.

Hyperparameter Tuning for Logistic Regression

Correlation Analysis

Create correlation matrices to check for multi-collinearity.

Hyperparameter Tuning for Random Forest

Hyperparameter Tuning for Gradient Boosting Classifier

• Hyperparameter Tuning for Support Vector Machine

EDA: Accident Distribution by Day or Night

• Plot accident distribution by sunrise and sunset
• Plot accident distribution by sunrise and sunset and for each city.

Model Evaluation XGBoost

Evaluate XGBoost over test dataset

EDA: Mean Duration

• Plot mean duration of the accident by severity
• Plot mean duration of the accident by severity for each city

Preprocess the dataset, compress it, upload to GitHub

Russ is on this!

I will have the final result either end of today or tomorrow morning and we can go over the changes I made and how to work with the compressed version. Our goal is to trim it down a lot from 1.3 Million observations to around 75 thousand. We will only use data from certain US Cities.

Consolidate the preprocessing into one .py file for easier use.

Consolidate the preprocessing into one .py file for easier use.

Hyperparameter Tuning XGBoost

Tune XGBoost to find the best hyperparameter on the validation dataset.

Create a branch titled 'Eonyeon'

Create a new branch of project 3 for the team member 'Eonyeon Jo'.

Model Evaluation Adaboost

Evaluate Adaboost on the test dataset.

Hyperparameter tuning for Multinomial Naive Bayes

• Hyperparameter tuning for Multinomial Naive Bayes

Model Evaluation: Multinomial Naive Bayes

• Model evaluation for the multinomial naive bayes

EDA: By year and city (Discrepancy from 16' to 20')

• Number of accidents by year overall and by cities. Note the discrepancy from 2016 to 2020

Upload the Project Proposal

Upload the project proposal to the doc folder.

Create EDA Folder under Docs

Create a new folder called "EDA" under the docs folder, which will carry all the images that we generated for our EDA.

EDA: Side accident

• Plot the distribution of accidents by side of street the accident occurred (left or right)

Distribution by city

Create a distribution of accident severity by city (to accompany ANOVA)

Hyperparameter Tuning for Naive Bayes

EDA: By month and weekday

• Distribution of accidents by month and weekday split by severity. We can also provide the stats by cities.

Preprocess Location + Basic Variables

In this issue, we plan to preprocess the following variables:

'ID', 'Severity', 'Start_Time', 'End_Time', 'Distance(mi)', 'Description', 'Start_Lat', 'Start_Lng', 'End_Lat', 'End_Lng', 'Number', 'Street', 'Side', 'City', 'County', 'State', 'Zipcode', 'Country', 'Timezone', 'Airport_Code'

The goal is to impute the missing values and to drop the irrelevant columns.

Hyperparameter Tuning for Adaboost

Tune Adaboost to find the best hyperparameter over the validation dataset.

Transform Project Proposal Using the ACM Template

Create project proposal on overleaf and share with the team.

Factor Analysis

Finding the factors that carry the greatest significance.

Data: Build Script to Generate Imputed Data

Add a detailed readme

Add a detailed readme, providing more details about the project.

Chi Square Test

Conduct analysis of variance by city to determine a trend in accident rate by city.

Model Evaluation: Gradient Boosting Classifier

• Model evaluation for the support vector machine

EDA: Side accident

• Plot the distribution of accidents by side of street the accident occurred (left or right)

Post Modeling Analysis

• Generate feature importance plots to understand what features were important for predicting the severity of the accidents.
• Generate learning curves.
• Generate probability distribution plots for prediction probabilities.

Preprocess Infrastructure

Infrastructure Analysis will conducted in this issues to preprocess the following variables:

'Traffic_Signal', 'Crossing', 'Station','Amenity', 'Bump', 'Give_Way', 'Junction', 'No_Exit', 'Railway', 'Roundabout','Stop', 'Traffic_Calming', 'Turning_Loop'.

Complete Project Proposal on Google Docs

Complete project proposal on google docs based on project requirements.

Commit an ML project template

Find an ML project template or any suitable template that we can use to manage the code in a standard way.