Monotonic-Optimal-Binning

Python implementation (MOBPY)

MOB is a statistical approach designed to transform continuous variables into categorical variables in a way that ensures both optimality and monotonicity. The project consists of two main modules: MOB and PAVA. The MOB module is responsible for achieving monotonic optimal binning, while the PAVA module utilizes the pool adjacent violators algorithm.

In MOB, we have extended the functionality to allow users to merge bins either based on statistics or bin size, giving them greater control over the binning process. On the other hand, PAVA offers flexibility by handling multiple statistics while MOB can only deal with average so far, and it also enables the display of other variables with given statistics in the final results.

This Python-based project empowers programmers to obtain precise and tailored discretization results, aligning with their specific expectations for monotonic categorical variables.

Installation

python3 -m pip install MOBPY

Usage

MOB Example:

import pandas as pd
from MOBPY.MOB import MOB


if __name__ == '__main__' :
    # import the testing datasets
    df = pd.read_csv('/data/german_data_credit_cat.csv')
    
    # Original values in the column are [1,2], make it into 1 representing the positive term, and 0 for the other one.
    df['default'] = df['default'] - 1

    # run the MOB algorithm to discretize the variable 'Durationinmonth'.
    MOB_ALGO = MOB(data = df, var = 'Durationinmonth', response = 'default', exclude_value = None)
    # A must-do step to set the binning constraints.
    MOB_ALGO.setBinningConstraints( max_bins = 6, min_bins = 3, 
                                    max_samples = 0.4, min_samples = 0.05, 
                                    min_bads = 0.05, 
                                    init_pvalue = 0.4, 
                                    maximize_bins=True)
    # execute the MOB algorithm.
    SizeBinning = MOB_ALGO.runMOB(mergeMethod='Size') # Run under the bins size base.

    StatsBinning = MOB_ALGO.runMOB(mergeMethod='Stats') # Run under the statistical base.

The runMOB method will return a pandas.DataFrame which shows the binning result of the variable and also the WoE summary information for each bin.

_{MOB algorithm binning result}

And after we receive the binning result dataframe, we can plot it by using MOBPY.plot.MOB_PLOT.plotBinsSummary to visualize the binning summary result.

from MOBPY.plot.MOB_PLOT import MOB_PLOT

# plot the bin summary data.
print('Bins Size Base')
MOB_PLOT.plotBinsSummary(monoOptBinTable = SizeBinning, var_name = 'Durationinmonth')

print('Statisitcal Base')
MOB_PLOT.plotBinsSummary(monoOptBinTable = StatsBinning, var_name = 'Durationinmonth')

_{MOB visualization example}

PAVA Example:

import pandas as pd
from MOBPY.PAVA import PAVA
from MOBPY.plot.MOB_PLOT import MOB_PLOT

#import testing dataset
df = pd.read_csv('/Users/chentahung/Desktop/git/mob-py/data/insurance2.csv')
# construct the PAVA object
P = PAVA(data = df, var = 'age', response = 'insuranceclaim', metric='mean', 
         add_var_aggFunc = {'bmi':'mean', 'smoker':'sum', 'region':['max', 'min'], 'charges' : np.ptp}, exclude_value = None)
P.runPAVA(sign = 'auto')

# There are four attributes that will be stored in the PAVA object after running `runPAVA`
print(P.OrgDataAssignment)
print(P.CSD_Summary)
print(P.GCM_Summary)
print(P.PAV_Summary)

_{PAVA.PAV_Summary output}

# Visualize the Cumulative Sum Diagram and Greatest Convex Minorant of the CSD
MOB_PLOT.plotPAVACsd(CSD_Summary = P.CSD_Summary)

# apply the PAVA result to a pandas.Series and return either a interval display or a value of starting or closing value of the interval range.
_pavRes = P.applyPAVA(df['age'], 'interval')

_{PAVA CSD & GCM visualization example}

Highlighted Features

User Preferences:

Monotonic Optimal Binning

The MOB algorithm offers two user preference settings (mergeMethod argument):

Size: This setting allows you to optimize the sample size of each bin within specified maximum and minimum limits while ensuring that the minimum number of bins constraint is maintained.
Stats: With this setting, the algorithm applies a stricter approach based on hypothesis testing results.

Typically, the 'Stats' (statistical-based) and 'Size' (bin size-based) methods yield identical results. However, when dealing with data under certain scenarios where the 'Size' method, employed by MOB, tends to prioritize maintaining the population of each bin within the maximum and minimum limits. In contrast, the 'Stats' method adheres to a more rigorous logic based on the results of hypothesis testing.

For example, following the previos example code.

# run the MOB algorithm to discretize the variable 'Creditamount'.
MOB_ALGO = MOB(data = df, var = 'Creditamount', response = 'default', exclude_value = None) 
# Set Binning Constraints (Must-Do!)
MOB_ALGO.setBinningConstraints( max_bins = 6, min_bins = 3, 
                                max_samples = 0.4, min_samples = 0.05, 
                                min_bads = 0.05, 
                                init_pvalue = 0.4, 
                                maximize_bins=True)
# mergeMethod = 'Size' means to run MOB algorithm under bins size base
SizeBinning = MOB_ALGO.runMOB(mergeMethod='Size')
StatsBinning = MOB_ALGO.runMOB(mergeMethod='Stats')

# plot the bin summary data.
print('Bins Size Base')
MOB_PLOT.plotBinsSummary(monoOptBinTable = SizeBinning)
print('Statisitcal Base')
MOB_PLOT.plotBinsSummary(monoOptBinTable = StatsBinning)

SizeBinning	StatsBinning
runMOB(mergeMethod='Size') (bins size base)	runMOB(mergeMethod='Stats') (statistical base)

The left side image is the result generated by mergeMethod = 'Size' (bin size-based), and the right side is the result generated by mergeMethod = 'Stats' (statistical-based). We can see that the 'Size' method is designed to merge bins that fail to meet the minimum sample population requirement. This approach ensures that the number of bins remains within the specified limit, preventing it from exceeding the minimum bin limitation. By merging bins that fall short of the population threshold, the 'Size' method effectively maintains a balanced distribution of data across the bins..

Extended Application:

Pool Adjacent Violators Algorithm

The PAVA module allows users to include additional variables in a dictionary format, along with their corresponding statistics metrics, which will be displayed in the final PAVA result table. Unlike the MOB module, which is limited to calculating the mean (average) for the metric, the PAVA module accepts multiple statistics options, valid inputs include 'count', 'mean', 'sum', 'std', 'var', 'min', 'max', 'ptp'. It's important to note that the add_var_aggFunc argument usage is similar to inputting a dictionary for different aggregations per column, referring to pandas.DataFrame.agg. See PAVA documentation for more information.

# construct the PAVA object
P = PAVA(data = df, var = 'age', response = 'insuranceclaim', metric='mean', 
         add_var_aggFunc = {'bmi':'mean', 'smoker':'sum', 'region':['max', 'min'], 'charges' : np.ptp}, exclude_value = None)

_{PAVA.PAV_Summary output}

Full Documentation

↪ Full API Reference

Environment

OS : macOS Ventura

IDE: Visual Studio Code 

Language : Python 3.9.7 
    - pandas 1.3.4
    - numpy 1.20.3
    - scipy 1.7.1
    - matplotlib 3.7.1
    - typing 3.7.4.3

Reference

Mironchyk, Pavel, and Viktor Tchistiakov. "Monotone optimal binning algorithm for credit risk modeling." Utr. Work. Pap (2017).
Smalbil, P. J. "The choices of weights in the iterative convex minorant algorithm." (2015).
Testing Dataset 1 : German Credit Risk from Kaggle
Testing Dataset 2 : US Health Insurance Dataset from Kaggle
GitHub Project : Monotone Optimal Binning (SAS 9.4 version)

Authors

Ta-Hung (Denny) Chen
- LinkedIn Profile : https://www.linkedin.com/in/dennychen-tahung/
- E-Mail : [email protected]
Yu-Cheng (Darren) Tsai
- LindedIn Profile : https://www.linkedin.com/in/darren-yucheng-tsai/
- E-Mail :
Peter Chen
- LinkedIn Profile : https://www.linkedin.com/in/peterchentsungwei/
- E-Mail : [email protected]

chentahung / monotonic-optimal-binning Goto Github PK