xydrolase/shap4j-data-converter

Converting tree ensemble model dumps as shap4j data files

shap4j-data-converter

Data converter for shap4j, which converts tree ensemble models trained by XGBoost, LightGBM, CatBoost, scikit-learn and pyspark to .shap4j data files.

Getting started

Installation

 $ python setup.py install

Dump tree model through pickle

XGBoost

Here, we borrow the example in the shap documentation

import xgboost
import shap
import pickle

# train XGBoost model
X,y = shap.datasets.boston()
model = xgboost.train({"learning_rate": 0.01}, xgboost.DMatrix(X, label=y), 100)

with open("boston.pkl", "wb+") as f:
    pickle.dump(model, f)

Convert an existing .pkl file with command-line tool

We can use the shap4jconv program to convert the boston.pkl pickle file generated by the previous example to a .shap4j data file:

 $ shap4jconv boston.pkl

The command above generates the output file as boston.shap4j

Alternatively, you could specify the output file via the --output parameter:

 $ shap4jconv --output model.shap4j --overwrite boston.pkl

where the --overwrite parameter allows shap4jconv to overwrite an existing file.

Convert .pkl file programatically

You can also use shap4j-data-converter in your Python program by simply importing the Shap4jDataConverter class from the shap4jconv package, for example:

from shap4jconv import Shap4jDataConverter

converter = Shap4jDataConverter()
converter.convert("boston.pkl", output_file="dumped_boston.pkl", overwrite=True)

Use .shap4j data file in Java

See the Java example of shap4j for how to integrate SHAP into your JVM projects using the data files generated above.

shap4j-data-converter's Issues

Conversion of tree models fails

I can't convert a tree model using shap4j

shap4jconv modelpkles.pkl Traceback (most recent call last): File "/Users/tpearson/personal/git/shap4j-data-converter/bin/./shap4jconv", line 5, in <module> from shap4jconv import Shap4jDataConverter File "<frozen zipimport>", line 259, in load_module File "/Users/tpearson/personal/git/shap4j-data-converter/venv/lib/python3.9/site-packages/shap4j_data_converter-0.0.2-py3.9.egg/shap4jconv/__init__.py", line 8, in <module> ModuleNotFoundError: No module named 'shap.explainers.tree'

shap import fails with current shap from pip

Hi, just wanted to say that the data converter sadly fails to execute
with the following message while current shap is installed from pip.
do i need to get a specific version or so to get it running?

bests,
Nikolaus

Traceback (most recent call last):
  File "/usr/local/bin/shap4jconv", line 4, in <module>
    __import__('pkg_resources').run_script('shap4j-data-converter==0.0.2', 'shap4jconv')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 667, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1470, in run_script
    exec(script_code, namespace, namespace)
  File "/usr/local/lib/python3.8/dist-packages/shap4j_data_converter-0.0.2-py3.8.egg/EGG-INFO/scripts/shap4jconv", line 5, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.8/dist-packages/shap4j_data_converter-0.0.2-py3.8.egg/shap4jconv/__init__.py", line 8, in <module>
ModuleNotFoundError: No module named 'shap.explainers.tree'```

Setup script fails with error numpy 1.26.0rc1 is installed but numpy<1.26,>=1.21 is required by {'numba'}

System: Mac OS X Ventura 13.5.2
Python: Python 3.9.0

Steps to reproduce

python3 -m venv venv
source venv/bin/activate
python setup.py install

I see
`Installed /Users/tpearson/personal/git/shap4j-data-converter/venv/lib/python3.9/site-packages/shap4j_data_converter-0.0.2-py3.9.egg
Processing dependencies for shap4j-data-converter==0.0.2
Searching for numpy
Reading https://pypi.org/simple/numpy/
Downloading https://files.pythonhosted.org/packages/d4/2f/f51bb073a4c787f9450f5942cf60378c899fb12463be1851c2c03fcd9537/numpy-1.26.0rc1-cp39-cp39-macosx_10_9_x86_64.whl#sha256=3080a9ec21470a9b485e92a09baedb5136468d89b2f2a1896a27fa9e36341af2
Best match: numpy 1.26.0rc1
Processing numpy-1.26.0rc1-cp39-cp39-macosx_10_9_x86_64.whl
Installing numpy-1.26.0rc1-cp39-cp39-macosx_10_9_x86_64.whl to /Users/tpearson/personal/git/shap4j-data-converter/venv/lib/python3.9/site-packages
Adding numpy 1.26.0rc1 to easy-install.pth file
Installing f2py script to /Users/tpearson/personal/git/shap4j-data-converter/venv/bin

Installed /Users/tpearson/personal/git/shap4j-data-converter/venv/lib/python3.9/site-packages/numpy-1.26.0rc1-py3.9-macosx-10.15-x86_64.egg`

Result: error:

numpy 1.26.0rc1 is installed but numpy<1.26,>=1.21 is required by {'numba'}