qbarthelemy/pypermut

Python package for permutation tests, for statistics and machine learning.

Pyꟼermut

PyPermut is a Python package implementing permutation tests, for statistics and machine learning.

PyPermut is distributed under the open source 3-clause BSD license.

Description

In a nutshell, this package extends some parametric and non-parametric statistical tests into permutation tests for multivariate samples, allowing the computation of exact p-values and the correction for multiple tests.

Futhermore, this package offers several functions for machine learning, like a generic permutation test for any metric based on the predictions of a model (for eg, AUROC, AUPR, negative log-likelihood, etc).

Permutations for statistical tests

This package implements several permutation tests (also called randomization tests).

Correlation tests:

• Pearson product-moment correlation r test,
• Spearman rank-order correlation ρ test.

Ordinal and numerical location tests:

• Student's t-test for related samples,
• Student's t-test for independent samples (and its adaptation into Welch's unequal variances t-test),
• Wilcoxon (signed-rank) T test,
• Mann-Whitney U test,
• one-way ANOVA for independent samples,
• Kruskal-Wallis H test,
• Friedman χ² test.

In permutation tests, the null distribution is sampled by random permutations of measurements. Making no hypothesis on the statistic distribution, the p-value is defined as the proportion of the null distribution with test statistic greater (or, for certain test statistic, lower) than or equal to the test statistic of the observed data. When all permutations are evaluated, these permutation tests give exact tests.

In case of several variables, they naturally perform a correction for multiple tests:

• rmax permutation test from Pearson r correlations,
• ρmax permutation test from Spearman ρ correlations,
• tmax permutation test from Student's t-tests for related samples,
• tmax permutation test from Student's t-tests for independent samples,
• Tmin permutation test from Wilcoxon T tests,
• Umin permutation test from Mann-Whitney U tests,
• Fmax permutation test from one-way ANOVAs for independent samples,
• Hmax permutation test from Kruskal-Wallis H tests,
• χmax permutation test from Friedman χ² tests.

In the context of correlated variables, they efficiently correct the family-wise error rate, being an alternative to the well-known Bonferroni correction which would lead to an undesirable lack of statistical power. These multiple variables can be useful to compute pairwise post-hoc tests.

Decision tree for statistical tests

How to use PyPermut

See examples.

References

Westfall and Young, Resampling-based multiple testing: examples and methods for p-value adjustment, Wiley-Blackwell, 1993.

Pesarin, Multivariate Permutation Tests – With Applications in Biostatistics, Wiley-Blackwell, 2001.

Nichols and Holmes, "Nonparametric permutation tests for functional neuroimaging: A primer with examples", Human Brain Mapping, vol 15, pp 1-25, 2001.

Edgington and Onghena, Randomization Tests, Fourth Edition, Chapman and Hall/CRC, 2007.

Documentation

TODO: put documentation on RTD.

Installation

From sources

To install PyPermut as a standard module:

pip install path/to/pypermut

To install PyPermut in editable / development mode, in the folder:

pip install poetry
poetry install

Testing

Use pytest.

Authors

Quentin Barthélemy, an anonymous researcher and Louis Mayaud, under the wisdom of Marco Congedo.

pypermut's Issues

Possibility of calculating the nonparametric permutation test and checking one-sided hypotheses (alternative greater and less)

How do I call the 'alternative' parameter at the Wilcoxon and Mannwhitneyu permutation tests? I checked the raw code and couldn't find this parameter in the implementation for both models. Am I missing something ?

Add p-min permutation tests

When the tests of the different variables have not the same degrees of freedom, it is not possible to estimate the joint null distribution using the max (or min) of the statistics.

In this case, a p-min test estimates the null distribution using the min of the p-values of the different variables.