mwolfca/pandas-profiling

Create HTML profiling reports from pandas DataFrame objects

Pandas Profiling

Generates profile reports from a pandas DataFrame. The pandas df.describe() function is great but a little basic for serious exploratory data analysis. pandas_profiling extends the pandas DataFrame with df.profile_report() for quick data analysis.

For each column the following statistics - if relevant for the column type - are presented in an interactive HTML report:

• Type inference: detect the types of columns in a dataframe.
• Essentials: type, unique values, missing values
• Quantile statistics like minimum value, Q1, median, Q3, maximum, range, interquartile range
• Descriptive statistics like mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness
• Most frequent values
• Histogram
• Correlations highlighting of highly correlated variables, Spearman, Pearson and Kendall matrices
• Missing values matrix, count, heatmap and dendrogram of missing values
• Text analysis learn about categories (Uppercase, Space), scripts (Latin, Cyrillic) and blocks (ASCII) of text data.
• File and Image analysis extract file sizes, creation dates and dimensions and scan for truncated images or those containing EXIF information.

Announcements

Version v2.8.0 released

News for users working with image datasets: pandas-profiling now has build-in supports for Files and Images. Moreover, the text analysis features have also been reworked, providing more informative statistics.

For a better feel, have a look at the examples section in the docs or read the changelog for a complete view of the changes.

Version v2.7.0 released

Performance

There were several performance regressions pointed out to me recently when comparing 1.4.1 to 2.6.0. To that end, we benchmarked the code and found several minor features introducing disproportionate computational complexity. Version 2.7.0 optimizes these, giving significant performance improvements! Moreover, the default configuration is tweaked for towards the needs of the average user.

Phased builds and lazy loading

A report is built in phases, which allows for new exciting features such as caching, only re-rendering partial reports and lazily computing the report. Moreover, the progress bar provides more information on the building phase and step.

Documentation

This version introduces more elaborate documentation powered by Sphinx. The previously used pdoc3 has been adequate initially, however misses functionality and extensibility. Several recurring topics are now documented, for instance the configuration parameters are documented and there are pages on big datasets, sensitive data, integrations and resources.

Support pandas-profiling

The development of pandas-profiling relies completely on contributions. If you find value in the package, we welcome you to support the project through GitHub Sponsors! It's extra exciting that GitHub matches your contribution for the first year.

Find more information here:

• Changelog v2.7.0
• Changelog v2.8.0
• Sponsor the project on GitHub

May 7, 2020 💘

---

Contents: Examples | Installation | Documentation | Large datasets | Command line usage | Advanced usage | Types | How to contribute | Editor Integration | Dependencies

---

Examples

The following examples can give you an impression of what the package can do:

• Census Income (US Adult Census data relating income)
• NASA Meteorites (comprehensive set of meteorite landings)
• Titanic (the "Wonderwall" of datasets)
• NZA (open data from the Dutch Healthcare Authority)
• Stata Auto (1978 Automobile data)
• Vektis (Vektis Dutch Healthcare data)
• Colors (a simple colors dataset)

Specific features:

• Russian Vocabulary (demonstrates text analysis)
• Cats and Dogs (demonstrates image analysis from the file system)
• Celebrity Faces (demonstrates image analysis with EXIF information)
• Website Inaccessibility (demonstrates URL analysis)
• Orange prices and Coal prices (showcases report themes)

Tutorials:

• Tutorial: report structure using Kaggle data (advanced) (modify the report's structure)

Installation

Using pip

You can install using the pip package manager by running

pip install pandas-profiling[notebook]

Alternatively, you could install the latest version directly from Github:

pip install https://github.com/pandas-profiling/pandas-profiling/archive/master.zip

Using conda

You can install using the conda package manager by running

conda install -c conda-forge pandas-profiling

From source

Download the source code by cloning the repository or by pressing 'Download ZIP' on this page. Install by navigating to the proper directory and running

python setup.py install

Documentation

The documentation for pandas_profiling can be found here. Previous documentation is still available here.

Getting started

Start by loading in your pandas DataFrame, e.g. by using

import numpy as np
import pandas as pd
from pandas_profiling import ProfileReport

df = pd.DataFrame(
    np.random.rand(100, 5),
    columns=["a", "b", "c", "d", "e"]
)

To generate the report, run:

profile = ProfileReport(df, title="Pandas Profiling Report")

Explore deeper

You can configure the profile report in any way you like. The example code below loads the explorative configuration file, that includes many features for text (length distribution, unicode information), files (file size, creation time) and images (dimensions, exif information). If you are interested what exact settings were used, you can compare with the default configuration file.

profile = ProfileReport(df, title='Pandas Profiling Report', explorative=True)

Learn more about configuring pandas-profiling on the Advanced usage page.

Jupyter Notebook

We recommend generating reports interactively by using the Jupyter notebook. There are two interfaces (see animations below): through widgets and through a HTML report.

This is achieved by simply displaying the report. In the Jupyter Notebook, run:

profile.to_widgets()

The HTML report can be included in a Jupyter notebook:

Run the following code:

profile.to_notebook_iframe()

Saving the report

If you want to generate a HTML report file, save the ProfileReport to an object and use the to_file() function:

profile.to_file("your_report.html")

Alternatively, you can obtain the data as json:

# As a string
json_data = profile.to_json()

# As a file
profile.to_file("your_report.json")

Large datasets

Version 2.4 introduces minimal mode. This is a default configuration that disables expensive computations (such as correlations and dynamic binning). Use the following syntax:

profile = ProfileReport(large_dataset, minimal=True)
profile.to_file("output.html")

Command line usage

For standard formatted CSV files that can be read immediately by pandas, you can use the pandas_profiling executable. Run

pandas_profiling -h

for information about options and arguments.

Advanced usage

A set of options is available in order to adapt the report generated.

• title (str): Title for the report ('Pandas Profiling Report' by default).
• pool_size (int): Number of workers in thread pool. When set to zero, it is set to the number of CPUs available (0 by default).
• progress_bar (bool): If True, pandas-profiling will display a progress bar.

More settings can be found in the default configuration file, minimal configuration file and dark themed configuration file.

Example

profile = df.profile_report(title='Pandas Profiling Report', plot={'histogram': {'bins': 8}})
profile.to_file("output.html")

Types

Types are a powerful abstraction for effective data analysis, that goes beyond the logical data types (integer, float etc.). pandas-profiling currently recognizes the following types:

• Boolean
• Numerical
• Date
• Categorical
• URL
• Path
• File
• Image

We have developed a type system for Python, tailored for data analysis: visions. Selecting the right typeset drastically reduces the complexity the code of your analysis. Future versions of pandas-profiling will have extended type support through visions!

Contributing

Read on getting involved in the Contribution Guide.

Editor integration

PyCharm integration

1. Install pandas-profiling via the instructions above

2. Locate your pandas-profiling executable.

   On macOS / Linux / BSD:

   $ which pandas_profiling
   (example) /usr/local/bin/pandas_profiling
   

   On Windows:

   $ where pandas_profiling
   (example) C:\ProgramData\Anaconda3\Scripts\pandas_profiling.exe
   

3. In Pycharm, go to Settings (or Preferences on macOS) > Tools > External tools

4. Click the + icon to add a new external tool

5. Insert the following values

   • Name: Pandas Profiling
   • Program: The location obtained in step 2
   • Arguments: "$FilePath$" "$FileDir$/$FileNameWithoutAllExtensions$_report.html"
   • Working Directory: $ProjectFileDir$

To use the PyCharm Integration, right click on any dataset file: External Tools > Pandas Profiling.

Other integrations

Other editor integrations may be contributed via pull requests.

Dependencies

The profile report is written in HTML and CSS, which means pandas-profiling requires a modern browser.

You need Python 3 to run this package. Other dependencies can be found in the requirements files:

Filename	Requirements
requirements.txt	Package requirements
requirements-dev.txt	Requirements for development
requirements-test.txt	Requirements for testing
setup.py	Requirements for Widgets etc.