syanga/pycit

(Conditional) Independence testing & Markov blanket feature selection using k-NN mutual information and conditional mutual information estimators. Supports continuous, discrete, and mixed data, as well as multiprocessing.

pycit

Framework for independence testing and conditional independence testing, with multiprocessing. Currently uses mutual information (MI) and conditional mutual information (CMI) as test statistics, estimated using k-NN methods. Also supports a routine for Markov blanket feature selection. Reports permutation-based p-values.

Installation

pip install pycit

Available Test Statistic Estimators

Mutual Information Estimators

• ksg_mi: k-NN estimator for continuous data
• bi_ksg_mi: "bias-improved" k-NN estimator for continuous data
• mixed_mi: k-NN estimator for discrete-continuous mixtures

Conditional Mutual Information Estimators

• ksg_cmi: k-NN estimator for continuous data
• bi_ksg_cmi: "bias-improved" k-NN estimator for continuous data
• mixed_cmi: k-NN estimator for discrete-continuous mixtures

Note: Also includes a differential entropy estimator: kl_entropy.

Example Usage

Independence Testing

from pycit import itest

# Test whether or not x and y are independent
pval = itest(x, y, test_args={'statistic': 'ksg_mi', 'n_jobs': 2})
is_independent = (pval >= 1.- confidence_level)

Conditional Independence Testing

from pycit import citest

# Test whether or not x and y are conditionally independent given z
pval = citest(x, y, z, test_args={'statistic': 'ksg_mi', 'n_jobs': 2})
is_conditionally_independent = (pval >= 1.- confidence_level)

Markov Blanket Feature Selection

from pycit.markov_blanket import MarkovBlanket

# specify CI test configuration
cit_funcs = {
    'it_args': {
        'test_args': {
            'statistic': 'ksg_mi',
            'n_jobs': 2
        }
    },
    'cit_args': {
        'test_args': {
            'statistic': 'ksg_cmi',
            'n_jobs': 2
        }
    }
}

# find Markov blanket of Y. x_data contains data from predictor variables, X_1,...,X_m
mb = MarkovBlanket(x_data, y_data, cit_funcs)
markov_blanket = mb.find_markov_blanket()

Dependencies:

• numpy
• scipy
• scikit-learn

References:

• Kozachenko, L. and Leonenko, N. (1987). Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii, 23(2):9–16.
• Kraskov, A., Stögbauer, H., and Grassberger, P. (2004). Estimating mutual information. Physical Review E, 69(6):066138.
• Frenzel, S. and Pompe, B. (2007). Partial mutual information for coupling analysis of multivariate time series. Physical Review Letters, 99(20):204101.
• Gao, W., Kannan, S., Oh, S., and Viswanath, P. (2017). Estimating mutual information for discrete-continuous mixtures. In NIPS'2017.
• Gao, W., Oh, S., and Viswanath, P. (2018). Demystifying fixed k-nearest neighbor information estimators. IEEE Transactions on Information Theory, 64(8):5629–5661.
• Runge, J. (2018). Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information. In AISTATS'18.
• Yang, A., Ghassami, A., Raginsky, M., Kiyavash, N., and Rosenbaum, E. (2020). Model-Augmented Estimation of Conditional Mutual Information for Feature Selection. In UAI'2020.

pycit's Issues

Why x_data is 3 dimensions ?

My data have 2 dimensions, data[0] means one sample, data[0][0] means first feature 's first data. i don't understand why x_data have 3 dimensions .
i can see, the program have
self.num_samples = x_data.shape[0]
self.num_features = x_data.shape[2]
so i know x_data have sample and feature, but another dimension is what ?
and i observed this x_data: predictor variables. shape (num_samples, x_dim, num_features) what is x_dim ?

Dataframe containing mixed data cannot be processed by itest().

I have a Numpy array containing mixed data (categorical, continuous). Inputting this into the itest() as

pval= itest(X, Y, test_args={'statistic': 'mixed_mi', 'n_jobs': 1})

brings me the error message

ValueError: could not convert string to float: 'NO'

from:

File [c:\XXX.venv\Lib\site-packages\sklearn\utils_array_api.py:185], in _asarray_with_order(array, dtype, order, copy, xp)
182 xp, _ = get_namespace(array)
183 if xp.name in {"numpy", "numpy.array_api"}:
184 # Use NumPy API to support order
--> 185 array = numpy.asarray(array, order=order, dtype=dtype)
186 return xp.asarray(array, copy=copy)
187 else:

How do I input mixed data in order to test it for independence?

which is the null hypothesis?

pval = itest(df.iloc[:,ix].values, df.iloc[:,jx].values, test_args={'statistic': 'mixed_mi', 'n_jobs': 8})
is_independent = (pval >= 1.- 0.95) #confidence_level = 0.95

So, the output is True. So, the two variables are independent.

Which is the null hypothesis here?