ploomber/sklearn-evaluation

Machine learning model evaluation made easy: plots, tables, HTML reports, experiment tracking and Jupyter notebook analysis.

Tip

Deploy AI apps for free on Ploomber Cloud!

Join our community | Newsletter | Contact us | Docs | Blog | Website | YouTube

Ploomber is the fastest way to build data pipelines ⚡️. Use your favorite editor (Jupyter, VSCode, PyCharm) to develop interactively and deploy ☁️ without code changes (Kubernetes, Airflow, AWS Batch, and SLURM). Do you have legacy notebooks? Refactor them into modular pipelines with a single command.

Installation

Compatible with Python 3.7 and higher.

Install with pip:

pip install ploomber

Or with conda:

conda install ploomber -c conda-forge

Getting started

Try the tutorial:

Community

• Join us on Slack
• Newsletter
• YouTube
• Contact the development team

Main Features

⚡️ Get started quickly

A simple YAML API to get started quickly, a powerful Python API for total flexibility.

get-started.mp4

⏱ Shorter development cycles

Automatically cache your pipeline’s previous results and only re-compute tasks that have changed since your last execution.

shorter-cycles.mp4

☁️ Deploy anywhere

Run as a shell script in a single machine or distributively in Kubernetes, Airflow, AWS Batch, or SLURM.

deploy.mp4

📙 Automated migration from legacy notebooks

Bring your old monolithic notebooks, and we’ll automatically convert them into maintainable, modular pipelines.

refactor.mp4

I want to migrate my notebook.

Show me a demo.

Resources

• Documentation
• PyData Chicago talk (covers motivation and demo)
• Develop and deploy an ML pipeline in 30 minutes (EuroPython 2021)
• Guest blog post on the official Jupyter blog
• Examples (Machine Learning pipeline, ETL, among others)
• Blog
• Comparison with other tools
• More videos

About Ploomber

Ploomber is a big community of data enthusiasts pushing the boundaries of Data Science and Machine Learning tooling.

Whatever your skillset is, you can contribute to our mission. So whether you're a beginner or an experienced professional, you're welcome to join us on this journey!

Click here to know how you can contribute to Ploomber.

sklearn-evaluation's Issues

Rename plots.confusion_matrix_ to plots.confusion_matrix_

Report module API is too verbose

Improve Report model so the constructor doesn't need all parameters. The class should be able to figure out what can it do based on the parameters passed either in the constructor or added later.

Add validation curve to plot module

add tests for tn_at and fn_at

Missing chart in docs

https://sklearn-evaluation.readthedocs.io/en/stable/user_guide/report.html

Please show the counts in the confusion matrix by default

In your current example of the confusion matrix:

it would be valuable to show the counts by default as well, as color does not effectively communicate exact values. The color is still useful for viewing general trends, however, so I think a combination of color + actual value would work best.

Add plot: Joint plot

We need to add a new plot which can draw the main diagram with each of the features in a histogram. See example

Error in `plot.grid_search` when parameters grid has a single parameter

Describe the bug

plot.grid_search throws a TypeError when the cv_results_ is obtained from varying a single parameter.

Steps/Code to Reproduce

import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn import datasets

from sklearn_evaluation.plot import grid_search

iris = datasets.load_iris()

parameters = {
   'n_estimators': [1, 10, 50, 100],
}

est = RandomForestClassifier()
clf = GridSearchCV(est, parameters, cv=5)

X, y = datasets.make_classification(1000, 10, n_informative=5, class_sep=0.7)
clf.fit(X, y)

# changing numeric parameter without any restrictions
# in the rest of the parameter set
grid_search(clf.cv_results_, change='n_estimators')
plt.show()

Expected Results

TypeError is thrown.

Actual Results

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Untitled-1 in <cell line: 20>()
     <a href='untitled:Untitled-1?line=15'>16</a> clf.fit(X, y)
     <a href='untitled:Untitled-1?line=16'>17</a> # changing numeric parameter without any restrictions
     <a href='untitled:Untitled-1?line=17'>18</a> # in the rest of the parameter set
---> <a href='untitled:Untitled-1?line=19'>20</a> grid_search(clf.cv_results_, change='n_estimators')
     <a href='untitled:Untitled-1?line=20'>21</a> plt.show()

File ~/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py:81, in grid_search(cv_results_, change, subset, kind, cmap, ax)
     [74](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=73)[ grid_scores = [
     ]()[75](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=74)[     gs(p, m, s) for p, m, s in
     ]()[76](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=75)[     zip(cv_results_['params'], cv_results_['mean_test_score'],
     ]()[77](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=76)[         cv_results_['std_test_score'])
     ]()[78](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=77)[ ]
     ]()[80](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=79)[ if isinstance(change, string_types) or len(change) == 1:
---> ]()[81](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=80)[     return _grid_search_single(grid_scores, change, subset, kind, ax)
     ]()[82](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=81)[ elif len(change) == 2:
     ]()[83](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=82)[     return _grid_search_double(grid_scores, change, subset, cmap, ax)

File ~/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py:142, in _grid_search_single(grid_scores, change, subset, kind, ax)
    ]()[140](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=139)[ label = ['{}: {}'.format(*t) for t in params_kv]
    ]()[141](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=140)[ # now convert it to one string
--> ]()[142](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=141)[ label = reduce(lambda x, y: x + ', ' + y, label)
    ]()[144](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=143)[ if kind == 'bar':
    ]()[145](file:///home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/sklearn_evaluation/plot/grid_search.py?line=144)[     bar_shifter(y, yerr=stds, label=label)

TypeError: reduce() of empty iterable with no initial value]()

Versions

import sklearn; sklearn.show_versions()

System:
    python: 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:39:04) [GCC 10.3.0]
executable: /home/adrien/mambaforge/envs/fno/bin/python
   machine: Linux-4.15.0-175-generic-x86_64-with-glibc2.27

Python dependencies:
          pip: 22.0.4
   setuptools: 62.1.0
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
/home/adrien/mambaforge/envs/fno/lib/python3.10/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

import sklearn_evaluation; sklearn_evaluation.__version__

'0.5.7'

Rename functions that take a percent/proportion parameter

Some functions in the metrics module take a percent parameter, but the computations uses as a proportion. The API must be consistent.

Make some requirements optional

Modules have different dependencies, the package should not force all requirements if user is only going to use one module.

Visualize results of grid search?

Do you plan to add support for visualizing the results of a sklearn GridSearch, or is that out of the scope of this project? You could use line charts to visualize one parameter at a time, and heatmaps to visualize two parameters at a time.

This binder example fails

https://sklearn-evaluation.readthedocs.io/en/latest/user_guide/compare.html

Add Model Comparison Functionality to NotebookCollection

Model comparability with the plot argument of a NotebookCollection object could be improved.

At the moment, models must be cycled through in order to compare plots. See documentation here for example. Perhaps it would benefit to add additional functionality that allows side by side comparisons, similar to that described in the advanced matplotlib usage guide here.

Things to consider: limitations on the number of models that can be plotted simultaneously (formatting issues), whether plotting models simultaneously is affected by window size, the degree to which the quality of individual plots is affected (if axes/labels/symbols get too small, perhaps we could reevaluate the inputs to improve visibility/model quality).

Add Feature Importance Tracking

If models are run in parallel, we should try and include additional visualizations for evaluating feature importances. Local and Global feature importances can be calculated using SHAP values. From my limited understanding, one thing that sklearn does not go into significant depth with interpretability/feature importances. Perhaps we could change that/implement some additional comparison metrics.

We can also try and incorporate specific visualizations if we use LASSO/ridge regression models. These are just some thoughts that popped into my head, but will comment below when I have fleshed out ideas my ideas.

Implement __setattr__ in trained models

These should raise an exception:

tm.plots.confusion_matrix = 1
tm.plots = 1

Do not assume the estimator has a estimators_ attribute

The function to plot feature_importances assumes the estimator has a estimators_ property. This is true for some (e.g. rf, adaboost) but not for every estimator (e.g. decision tree). The function fails if called on a decision tree and that should not be the case.

new plots

we only have evaluation plots for ML model classifiers, but we should add more; for example, we could create module regression and clustering plots. we can draw some inspiration from these two projects:

https://github.com/reiinakano/scikit-plot

https://github.com/DistrictDataLabs/yellowbrick

cc @idomic

Grid_score -> cv_result

Starting from sklearn 0.2 grid_score is not available anymore... instead they use cv_result, would be nice to adapt the code to work with this one,

Thanks for your work

Romain

grid_search error log should be more specific

The exception log is saying som_param_rf_n_estimators is not a valid parameter, it should imply what are the valid parameters and how the user can move on with the evaluation.

Add new module: clustering

We need to create a module for regression plots (can leverage the existing layout from classification)

Add testing cases for validation and learning curve

Add new module: Regression

We need to create a module for regression plots (can leverage the existing layout from classification)

plot did not work

hi @edublancasx
This repo is very useful for me ! Great work !
But I meet a very strange problem when I getting start to use sklearn-evaluation.

I run the code as below to show the confusion matrix:
#code
...
from sklearn_evaluation import plot
...
plot.confusion_matrix(y_true, y_pred, target_names=target_names)
...

But the plot window was not created .And there was not any error log .I'm sure the 'y_true' and 'y_pred' are both correct.

am I missing something?（sth like calling 'show()' etc?）
Thank you very much!

Add learning curve to plots module

move to scikit-learn contrib?

We have a scikit-learn-contrib organization now that could open this to a wider audience:
https://github.com/scikit-learn-contrib/scikit-learn-contrib/blob/master/README.md

Make package backend agnostic

Do not assume any type of matplotlib backend on the code. Use pyplot, which automatically detects backends available. Plot module is backend agnostic now, but the report module is not (and imports the Agg backend).

Update readme to explain the metrics module

Remove metrics module

The original codebase for this project contained a metrics module which is still present. Since this project is focused on visualization, such module is no longer relevant and should be removed.

Add lift plot

http://blog.datalifebalance.com/lift-charts-a-data-scientists-secret-weapon/

Add links to download notebook and train.py in notebook collection tutorial

Migrate tests from nox to tox

Tox supports conda

Incompatibility with sklearn 0.20.0

Hi.
I was trying to use this package with the up-to-dated version of scikit (0.20.0) but I did not understand how to do it. In particular, I was trying to use

from sklearn_evaluation import plot
plot.grid_search(gridCV.grid_scores_, change=change,kind='bar')

but the member grid_scores_ does not exist any more (present till scikit 0.17) and has been substituted by cv_results_, which returns an object of different data type with respect to the former member.
Is there an easy way to go on using this function by using the new cv_results_ in place of grid_scores_?
Thank you.

Improve Precision-Recall and ROC curve performance

Precision-Recall and ROC curve use label_binarize each time they are called.

Plots module should expose matplotlib interface

Plots module should let users tune the graph using the matplotlib interface (e.g. changing colors, labels, etc)

Make sure CI fails if ipython directives in docs raise an exception

Add Information Value and decile analysis using Lift Chart

It will be a lot more helpful for fellow data scientist, if these two functionalities can be added.

Information value:
http://ucanalytics.com/blogs/information-value-and-weight-of-evidencebanking-case/

Report generation broken in Python 3

Version 0.2 breaks when generating reports from Python 3 due to the differences in the StringIO class.

GridSearch heatmap for 'None' parameter

When I try to generate a heatmap for GridSearchCV results, if the parameter has 'None' type, it gives error:
TypeError: '<' not supported between instances of 'NoneType' and 'int'

The parameter can be, for e.g.
max_depth_for_decision_trees = [3, 5, 10, None].

Is there any workaround for this?

Fix report testing

Fix imports to follow PEP 328

https://www.python.org/dev/peps/pep-0328/

Make the plots module API consistent

Right now, precision_at_proportions has a ax parameter. This makes easy to plot many curves one over another, but the rest of the functions do not have such parameter and also return a matplotlib Figure object instead of a Axes object, API must be consistent.

`plot.grid_search` throws a `TypeError` when parameters cannot be sorted.

Describe the bug

plot.grid_search throws a TypeError when parameters cannot be sorted.

Steps/Code to Reproduce

import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn import datasets

from sklearn_evaluation.plot import grid_search

iris = datasets.load_iris()

std = StandardScaler()
est = KNeighborsClassifier(n_neighbors=5)

pipe = Pipeline([
   ('std', std),
   ('est', est)
])

parameters = {
  'std': [None, std],
  'est__n_neighbors': [1, 10, 50, 100],
}

clf = GridSearchCV(pipe, parameters, cv=5)

X, y = datasets.make_classification(1000, 10, n_informative=5, class_sep=0.7)
clf.fit(X, y)

# changing numeric parameter without any restrictions
# in the rest of the parameter set
grid_search(clf.cv_results_, change='est__n_neighbors')
plt.show()

Expected Results

Should display Grid search results plot with legend :

std: None
std: StandardScaler()

Actual Results

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/adrien/Dropbox/workspace/sklearn-evaluation/issues/issue_43.py in <cell line: 23>()
     [30](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/issues/issue_43.py?line=29) clf.fit(X, y)
     [32](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/issues/issue_43.py?line=31) # changing numeric parameter without any restrictions
     [33](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/issues/issue_43.py?line=32) # in the rest of the parameter set
---> [34](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/issues/issue_43.py?line=33) grid_search(clf.cv_results_, change='est__n_neighbors')
     [35](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/issues/issue_43.py?line=34) plt.show()

File ~/Dropbox/workspace/sklearn-evaluation/issues/../src/sklearn_evaluation/plot/grid_search.py:81, in grid_search(cv_results_, change, subset, kind, cmap, ax)
     [74](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=73) grid_scores = [
     [75](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=74)     gs(p, m, s) for p, m, s in
     [76](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=75)     zip(cv_results_['params'], cv_results_['mean_test_score'],
     [77](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=76)         cv_results_['std_test_score'])
     [78](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=77) ]
     [80](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=79) if isinstance(change, string_types) or len(change) == 1:
---> [81](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=80)     return _grid_search_single(grid_scores, change, subset, kind, ax)
     [82](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=81) elif len(change) == 2:
     [83](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=82)     return _grid_search_double(grid_scores, change, subset, cmap, ax)

File ~/Dropbox/workspace/sklearn-evaluation/issues/../src/sklearn_evaluation/plot/grid_search.py:132, in _grid_search_single(grid_scores, change, subset, kind, ax)
    [126](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=125)     # bar shifter is just a wrapper around matplotlib's bar plot
    [127](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=126)     # to automatically calculate the left position on each bar
    [128](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=127)     bar_shifter = BarShifter(g_number=change_unique,
    [129](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=128)                              g_size=len(groups),
    [130](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=129)                              ax=ax)
--> [132](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=131) for params_kv, group in _sorted_map_iter(groups):
    [133](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=132)     # get the x and y values for each grid_score on this group
    [134](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=133)     # also calculate the std
    [135](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=134)     x = [element.parameters[change] for element in group]
    [136](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/plot/grid_search.py?line=135)     y = [element.mean_validation_score for element in group]

File ~/Dropbox/workspace/sklearn-evaluation/issues/../src/sklearn_evaluation/util.py:85, in _sorted_map_iter(d)
     [84](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/util.py?line=83) def _sorted_map_iter(d):
---> [85](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/util.py?line=84)     ord_keys = sorted(d.keys())
     [86](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/util.py?line=85)     for k in ord_keys:
     [87](file:///home/adrien/Dropbox/workspace/sklearn-evaluation/src/sklearn_evaluation/util.py?line=86)         yield (k, d[k])

TypeError: '<' not supported between instances of 'StandardScaler' and 'NoneType'

Versions

import sklearn; sklearn.show_versions()

System:
    python: 3.9.10 | packaged by conda-forge | (main, Feb  1 2022, 21:24:11)  [GCC 9.4.0]
executable: /home/adrien/mambaforge/bin/python
   machine: Linux-4.15.0-175-generic-x86_64-with-glibc2.27

Python dependencies:
          pip: 22.0.4
   setuptools: 60.10.0
      sklearn: 1.0.2
        numpy: 1.22.3
        scipy: 1.8.0
       Cython: None
       pandas: 1.4.2
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
/home/adrien/mambaforge/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

Plot: Add slihouette plot

We need to add the plot and score

Installing sklearn_evaluation

I used "pip install sklearn-evaluation" to install this library in anaconda. All requirements exit but it does not install. When I want to import it, there is no library.
When I run pip command to install it, it does not access to install, nor install anything.

Add plot: bi-dimentional feature importance

We already have a plot for 1 rank feature importance.

A two-dimensional ranking of features takes into account pairs of features at a time - joint plot analysis. The pairs of features are then ranked by score and visualized using the lower left triangle of a feature co-occurrence matrix.

New stable release

• remove circleci badge
• update docs url https://sklearn-evaluation.readthedocs.io/en/stable/
• delete gh-pages branch

Improve testing suite

I'm using matplotlib's testing functions for comparing the results of the plots module.

Due to the nature of the tests (image comparison), there are some problems (especially when running them in Travis). Although there are workarounds, some people (seaborn creator) recommend avoiding this type of testing

It seems a better idea to check the Axes object returned by every function instead of comparing them with baseline images. Most of the functions are actually computing metrics (and then putting results in the Axes object), I can grab come inspiration from the metrics tests in scikit-learn

Target names in confusion matrix should be optional

Fix travis

Use conda to avoid scipy install errors

Functionality: recursive feature elimination

We can take the users' features and eliminate the weakest features.
Example