sagyome/xgboosttreeapproximator

This project implements a method that converts a trained XGBoost classification tree into a single decision tree.

XGBoostTreeApproximator

This project implements a method that converts a trained XGBoost into a single decision tree. At this stage, this package is ideal for binary classification challenges. It can be applied for challenges with a few classes (2-3) but is not recommended for numerous classes, at least for the current implementation. Please see the included .ipynb file for further explanations of the API.

xgboosttreeapproximator's Issues

Hyper-parameter space of XGBoost

Hi,

I am trying to test the algorithm using the German dataset.
However, I have done XGBoost hyper-parameters search using the following code, but I cannot confirm the AUC (84.48±1.75) of the paper. The best model I could confirm was AUC = 81.56%. I have changed cv=3 to cv=10 according to the paper.

If there are any other changes needed, I would appreciate it if you could let me know.
Best regards,

XGBoostTreeApproximator/utils.py

Lines 74 to 97 in b378d9a

	def train_xgb_classifier(train,feature_cols,label_col,xgb_params):
	"""
	Train an XGBoost to the input dataframe
	:param train: pandas dataframe
	:param feature_cols: feature column names
	:param label_col: label column name
	:param xgb_params: Dict of XGBoost parameters
	:return: label column namened XGboost
	"""
	np.random.seed(RANDOM_SEED)
	tuning_params = {'colsample_bytree': [0.3,0.5,0.9],
	'learning_rate': [0.01,0.1],
	'max_depth': [2,5,10],
	'alpha': [1,10],
	'n_estimators':[50,100]}
	if train[label_col].nunique() > 2:
	xgb_params['objective'] = "multi:softprob"
	else:
	xgb_params['objective'] = "binary:logitraw"
	model = xg.XGBClassifier(xgb_params)
	clfGS = GridSearchCV(model, tuning_params, cv=3)
	clfGS.fit(train[feature_cols], train[label_col])
	return clfGS.best_estimator_

Can I generate RFBT and GBMBT, and save the BT models as files

I appreciate the great efforts that the authors have made in the paper "Approximating XGBoost with an interpretable decision tree".
I found this repo from this paper.
This repo has an ipynb file showing that a XGBoost based tree (XGBBT) can be fitted for the XGBoost model.

After trying the ipynb, I am thinking
(1) if there is a way to save the XGBBT as a file of the model?
(2) can I generate RFBT and GBMBT for RF and GBM like the conclusion mentioned in the paper?

Thank you.

error happened when pruning

def predict_probas_tree(self,conjunctions,X):
"""
Predict probabilities for X using a tree, represented as a conjunction set

    :param conjunctions: A list of conjunctions
    :param X: numpy array of data instances
    :return: class probabilities for each instance of X
    """

    probas = []
    for inst in X:
        for conj in conjunctions:
            if conj.containsInstance(inst):
                probas.append(conj.label_probas)
    return np.array(probas)

the length of return probas not equal to length of X, then the error happened when get_auc:

selected_indexes = [np.argmax([get_auc(Y,trees_predictions[i]) for i in trees_predictions])]

AttributeError: 'NoneType' object has no attribute 'groupdict'

I am using python 3.8. When I try to fit a FBT tree from a XGBoost model on the tic tac toe dataset I get an NoneType error on line 23 of TreesExtraction.py.

After analyzing the regex in feature_regex it seems to only match strings of the form 'feature name<number' or only continous features. Since the tic tac toe dataset uses only categorical features and has features with the form 'feature name=category' the regex does not match anything giving a NoneType error when calling groupdict() on the result.

It seems that the current codebase only supports datasets with continuous features. I tested this by also trying to fit a FBT tree on the Banknote dataset which only uses continuous features and it runs without any problems.

Difference between pseudo code and implementation

I think there is a difference between the pseudo code and the implementation in the calculation of IG in Algorithm 3 (A recursive splitting algorithm. The recursive call is referred as split) in the paper.
Calculating IG from Equation 4 in the paper, we consider that the code for the condition (IGj > IG) in line 11 of the pseudo code of Algorithm 3 is as follows.
if temp_entropy > best_entropy...
However, the implementation is as follows.
if temp_entropy < best_entropy...
Which is correct, the way IG is calculated in the paper (equation 4) or the implementation?
Relevant part
Best regards