learn-co-students/dsc-data-cleaning-project-dc-ds-career-042219

Project - Data Cleaning

Introduction

In this lab, we'll make use of everything we've learned about pandas, data cleaning, and exploratory data analysis. In order to complete this lab, you'll have to import, clean, combine, reshape, and visualize data to answer questions provided, as well as your own questions!

Objectives

You will be able to:

• Use different types of joins to merge DataFrames
• Identify missing values in a dataframe using built-in methods
• Evaluate and execute the best strategy for dealing with missing, duplicate, and erroneous values for a given dataset
• Inspect data for duplicates or extraneous values and remove them

The dataset

In this lab, we'll work with the comprehensive Super Heroes Dataset, which can be found on Kaggle!

Getting Started

In the cell below:

• Import and alias pandas as pd
• Import and alias numpy as np
• Import and alias seaborn as sns
• Import and alias matplotlib.pyplot as plt
• Set matplotlib visualizations to display inline in the notebook

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

For this lab, our dataset is split among two different sources -- 'heroes_information.csv' and 'super_hero_powers.csv'.

Use pandas to read in each file and store them in DataFrames in the appropriate variables below. Then, display the .head() of each to ensure that everything loaded correctly.

heroes_df = None
powers_df = None

It looks as if the heroes information dataset contained an index column. We did not specify that this dataset contained an index column, because we hadn't seen it yet. Pandas does not know how to tell apart an index column from any other data, so it stored it with the column name Unnamed: 0.

Our DataFrame provided row indices by default, so this column is not needed. Drop it from the DataFrame in place in the cell below, and then display the head of heroes_df to ensure that it worked properly.

Familiarize yourself with the dataset

The first step in our Exploratory Data Analysis will be to get familiar with the data. This step includes:

• Understanding the dimensionality of your dataset
• Investigating what type of data it contains, and the data types used to store it
• Discovering how missing values are encoded, and how many there are
• Getting a feel for what information it does and doesn't contain

In the cell below, get the descriptive statistics of each DataFrame.

Dealing with missing values

Starting in the cell below, detect and deal with any missing values in either DataFrame. Then, explain your methodology for detecting and dealing with outliers in the markdown section below. Be sure to explain your strategy for dealing with missing values in numeric columns, as well as your strategy for dealing with missing values in non-numeric columns.

Note that if you need to add more cells to write code in, you can do this by:

1. Highlighting a cell and then pressing ESC to enter command mode.
2. Press A to add a cell above the highlighted cell, or B to add a cell below the highlighted cell.

Describe your strategy below this line:

---

Joining, Grouping, and Aggregating

In the cell below, join the two DataFrames. Think about which sort of join you should use, as well as which columns you should join on. Rename columns and manipulate as needed.

HINT: Consider the possibility that the columns you choose to join on contain duplicate entries. If that is the case, devise a strategy to deal with the duplicates.

HINT: If the join throws an error message, consider setting the column you want to join on as the index for each DataFrame.

In the cell below, subset male and female heroes into different dataframes. Create a scatterplot of the height and weight of each hero, with weight as the y-axis. Plot both the male and female heroes subset into each dataframe, and make the color for each point in the scatterplot correspond to the gender of the superhero.

Some Initial Investigation

Next, slice the DataFrame as needed and visualize the distribution of heights and weights by gender. You should have 4 total plots.

In the cell below:

• Slice the DataFrame into separate DataFrames by gender
• Complete the show_distplot() function. This helper function should take in a DataFrame, a string containing the gender we want to visualize, and the column name we want to visualize by gender. The function should display a distplot visualization from seaborn of the column/gender combination.

Hint: Don't forget to check the seaborn documentation for distplot if you have questions about how to use it correctly!

male_heroes_df = None
female_heroes_df = None

def show_distplot(dataframe, gender, column_name):
    pass

# Male Height

# Male Weight

# Female Height

# Female Weight

Discuss your findings from the plots above, with respect to the distribution of height and weight by gender. Your explanation should include a discussion of any relevant summary statistics, including mean, median, mode, and the overall shape of each distribution.

Write your answer below this line:

---

Sample Question: Most Common Powers

The rest of this notebook will be left to you to investigate the dataset by formulating your own questions, and then seeking answers using pandas and numpy. Every answer should include some sort of visualization, when appropriate. Before moving on to formulating your own questions, use the dataset to answer the following questions about superhero powers:

• What are the 5 most common powers overall?
• What are the 5 most common powers in the Marvel Universe?
• What are the 5 most common powers in the DC Universe?

Analyze the results you found above to answer the following question:

How do the top 5 powers in the Marvel and DC universes compare? Are they similar, or are there significant differences? How do they compare to the overall trends in the entire Superheroes dataset?

Write your answer below this line:

---

Your Own Investigation

For the remainder of this lab, you'll be focusing on coming up with and answering your own question, just like we did above. Your question should not be overly simple, and should require both descriptive statistics and data visualization to answer. In case you're unsure of what questions to ask, some sample questions have been provided below.

Pick one of the following questions to investigate and answer, or come up with one of your own!

• Which powers have the highest chance of co-occurring in a hero (e.g. super strength and flight), and does this differ by gender?
• Is there a relationship between a hero's height and weight and their powerset?
• What is the distribution of skin colors amongst alien heroes?

Explain your question below this line:

---

Some sample cells have been provided to give you room to work. If you need to create more cells, you can do this easily by:

1. Highlighting a cell and then pressing esc to enter command mode.
2. Pressing b to add a cell below the currently highlighted cell, or a to add one above it.

Be sure to include thoughtful, well-labeled visualizations to back up your analysis!

Summary

In this lab, we demonstrated our mastery of:

• Using all of our Pandas knowledge to date to clean the dataset and deal with null values
• Using Queries and aggregations to group the data into interesting subsets as needed
• Using descriptive statistics and data visualization to find answers to questions we may have about the data