alcampopiano/hypothesize

Robust statistics in Python

Hypothesize

A Python package for hypothesis testing using robust statistics

Basic Example

A robust measure of association with winsorized correlation

from hypothesize.measuring_associations import wincor
from hypothesize.utilities import create_example_data

# creating an example DataFrame with columns "cell_1" and "cell_2"
df=create_example_data(2)

results=wincor(df.cell_1, df.cell_2)

# returning the correlation, number of observations, p-value, and winsorized covariance
print(results)
{'cor': 0.11, 'nval': 50, 'sig': 0.44, 'wcov': 0.01}

Documentation

📖 Please visit the Hypothesize documentation site. Note that each statistical test in the can be launched directly in Deepnote's hosted notebook environment—complete with sample data (as shown in the example above 👆).

Citing Hypothesize

If you use Hypothesize in academic work, please use the following citation:

Campopiano, A., & Wilcox, R. R. (2020). Hypothesize: Robust Statistics for Python. Journal of Open Source Software, 5(50), 2241, https://doi.org/10.21105/joss.02241

BibTex:

@article{Campopiano2020,
  doi = {10.21105/joss.02241},
  url = {https://doi.org/10.21105/joss.02241},
  year = {2020},
  publisher = {The Open Journal},
  volume = {5},
  number = {50},
  pages = {2241},
  author = {Allan Campopiano and Rand R. Wilcox},
  title = {Hypothesize: Robust Statistics for Python},
  journal = {Journal of Open Source Software}
}

hypothesize's Issues

function wish list following the 1.0 release

one group

• onesampb 1-alpha percentile boot CI for any estimator
• trimpb percentile boot CI for trimmed mean
• trimcibt bootstrap-t CI for trimmed mean
• mestci CI for M-measure of location based on huber's psi using percentile boot method (might be redundant with onesampb)
• momci CI for modified one-step M-estimator (might be redundant with onesampb)

two groups

• yuen yuen-welch method to compare trimmed means (no bootstrap)
• yuenbt bootstrapped-t CI for ut1 - ut2
• yhbt seems to be similar to yuenbt but modified for when trimming is <20 (maybe not needed)
• pb2gen percentile bootstrap CI for difference between any estimators
• m2ci convenience function func for comparing M-estimators based on huber's psi
• comvar2 bootstrapped comparison of variances
• permg permutation bootstrap test, any measure of location of scale
• t1way non-bootstrap method (but robust) for J indep groups (could be used for J>2 too)
• t1wayv2 same as t1way but explanatory es is returned for all pairs of groups

two dependent groups

• ydbt bootstrap-t CI for ut1 - ut2
• loc2dif difference between estimators using all combinations of difference scores
• l2drmci significance test for loc2dif using percentile bootstrap
• bootdpci percentile bootstrap method any estimator; can set options for using difference scores or measures of location based on the marginal distributions
• pcorb comparing variance of dep groups by extending some correlation-related method (i.e., pcorb(col1 - col1, col1 - col2) )
• pcorhc4 similar to pcorb; need more information on usage
• dfried some distance based test for J dependant groups (also used for more than 2 dep groups)

one-way for independent groups

• t1way non-bootstrap method (but robust) for J indep groups (could be in two indep group section too)
• t1wayv2 same as t1way but explanatory es is returned as well
• box1way another J=> 2 method based on trimmed means
• t1waybt test hyp of equal trimmed means using bootstrap t method (related to btrim which returns explanatory effect size and allows one to structure data a bit differently; btrim may not be needed)
• b1way percentile boot method for comparing J groups; seeing how deeply nest 0 is (1st method)
• other methods, especially ones using percentile bootstrap, under "methods based on MCP and linear contrasts" may be applicable here too

one-way methods based on multiple comparisons and linear contrasts

• lincon test linear contrasts with t means
• linconb test linear contrasts using bootstrap-t method
• tmcppb rom/hoch/ben-type methods using percentile bootstrap and trimmed means
• pbdepth percentile boot method for comparing J groups; seeing how deeply nest 0 is (2nd method)

two-way designs based on trimmed means

• t2way (no bootstrapping)

three-way designs based on trimmed means

• t3way (no bootstrapping)

two- and three-way multiple comparisons using contrasts (I believe for independent groups)

• mcp2atm all pairwise comparisons for each factor and interactions
• mcp3atm all pairwise comparisons for each factor and interactions
• bbtrim use bootstrap-t method for comparisons using contrasts
• bbbtrim use bootstrap-t method for comparisons using contrasts
• bbmcppb two-way percentile boot and trimmed mean tests
• bbbmcppb three-way percentile boot and trimmed mean tests

one-way dependant groups

• dfried some distance based test for J dependant groups
• rmanova trimmed means, no bootstrapping, for J groups
• rmmcp mcp for dep groups with trimmed means and Rom's method for FWE (might be able to extend to higher-level designs; 2 & 3-way)
• rmanovab bootstrap-t method for comparing measure associated with marginal distributions
• pairepb bootstrap-t method for all multi-comparisons
• bptd CI for all linear contrasts (very similar to pairdbp; but you can specify certain contrasts)
• bd1way percentile boot for J dep groups
• ddep another percentile boot method for J dep groups
• rmdzero percentile boot method for J group based on diff scores
• rmmcppb multiple comparisons for J dep groups using percentile boot method
• lindepbt boot-t method for mcp among J dep groups

within-within (two-way) dependent groups

• wwtrim non-bootstrap for trimmed means
• wwtrimbt same as wwtrim but bootstrap-t used
• wwmcp multi comps for main effects and interactions with linear contrasts (no boot)
• wwmcppb like wwmcp but percentile boot is used
• wwmcpbt like wwmcpppb but uses bootstrap-t method instead

mixed designs

• bwtrim no bootstrapping
• tsplitbt bootstrap-t for mixed design
• bwtrimbt same as tsplitbt but reports p values
• sppba test for factorA using percentile boot
• sppbb test for factorB using percentile boot
• sppbi test for interaction using percentile boot
• bwmcp all main effects and interactions for bw design bootstrap-t tests
• bwamcp same for factorA
• bwbmcp same for factorB
• bwimcp for interaction (non-bootstrap)
• spmcpa FA; same but with percentile boostrap
• spmcpb FB; same but with percentile boostrap
• spmcpi interaction; same but with percentile boostrap
• bwmcppb only for trimmed means? ; all main effects and interactions with percentile bootstrap method

three-way designs with one or more dependent groups

• bbwtrim no boot ominbus for main effect and interactions
• bwwtrim same as above two are within
• wwwtrim same as above all within
• bbwtrimbt no boot ominbus for main effect and interactions (bootstrap-t)
• bwwtrimbt same as above two are within (bootstrap-t)
• wwwtrimbt same as above all within (bootstrap-t)

three-way methods using multiple comparisons

• rm3mcp no bootstrap all contrasts
• bbwmcp bootstrap-t all comparisons with trimmed means
• bwwmcp bootstrap-t for the corresponding design
• bbwmcppb using percentile boot
• bwwmcppb using percentile boot
• wwwmcppb using percentile boot

effect sizes

• akp.effect delta (using trimmed mean and winsorized variance)
• yuenv2 compare two trimmed means and return explanatory effect size (xi2)
• ees.ci CI for two groups using percentile bootstrap method computes |xi|
• esmcp explanatory effect size returned for all pairs of J groups (can be used for dep groups)
• ESmainMCP a two-way method for getting explanatory effect size for FA and then FB
• esImcp two-way explanatory effect for all interactions

correlations and test of independence

• pbcor percentage bend correlation
• pball for a set of variables
• wincor winsorized correlation
• winall for a set of variables
• corb test for zero correlation using bootstrapping
• twopcor get CI of rho1 - rho2 (CI for difference of correlations) using percentile boot
• twocor test that two cors are equal (returns a p value and CI)

robust regression

• lsfitci CIs for reg parameters using percentile bootstrap method
• hc4wtest tests hypo that all slope parameters are zero using wild bootstrap method

utilities

• con1way create linear contrasts
• con2way
• con3way

Consistency with WRS2

Hi,

Is this package in sync with WRS2 v1.1-5?

Thanks

Chris

Functionality issues - JOSS review

This issue outlines some minor code issues in Hypothesize as part of the review for JOSS.

1. There is an error with the tr keyword argument of the trim_mean() function when using it as described in the API documentation. For example, when running trim_mean(df, tr=.2) I get:

TypeError: trim_mean() got an unexpected keyword argument 'tr' 

I believe this happens because Hypothesize is directly using the trim_mean() method from scipy whose keyword argument in this case is proportiontocut and not tr.

2. When using functions like bootdpci that take a list of positional arguments (*args), those parameters documented before the *args list cannot be passed as keyword arguments. In the case of bootdpci, for example, x and est cannot be passed as keyword arguments since tr cannot be passed as a keyword argument.

If for both point 1) and 2) the intention is that users of Hypothesize only specify parameters using positional arguments, it would be useful to have this requirement explicit somewhere in the documentation.

Not necessarily for this review but for a later stage in the development of Hypothesize, it would be a nice addition to add tests that assert that keyword arguments are behaving as expected.

test_corb fails

When running tests the test_corb function fails with the message:

E       AssertionError: assert {'ci': [-0.3537591633617352, 0.2968061934534894],\n 'cor': -0.07149960819769877,\n 'p_value': 0.5800000000000001} == {'ci': [-0.35375916336173524, 0.2968061934534894],\n 'cor': -0.07149960819769878,\n 'p_value': 0.5800000000000001}
E         Common items:
E         {'p_value': 0.5800000000000001}
E         Differing items:
E         {'cor': -0.07149960819769877} != {'cor': -0.07149960819769878}
E         {'ci': [-0.3537591633617352, 0.2968061934534894]} != {'ci': [-0.35375916336173524, 0.2968061934534894]}
E         Full diff:
E           {
E         -  'ci': [-0.35375916336173524, 0.2968061934534894],
E         ?                            -
E         +  'ci': [-0.3537591633617352, 0.2968061934534894],
E         -  'cor': -0.07149960819769878,
E         ?                            ^
E         +  'cor': -0.07149960819769877,
E         ?                            ^
E            'p_value': 0.5800000000000001,
E           }

It looks like this error is being thrown most likely due to slight version differences, numerical roundoff, or differences in machine architectures since the actual values are very close. Perhaps instead of looking for absolute equality these tests could use something like np.allclose?

increasing support level to Python 3.11

I saw the talk on hypothesize at PyCon, am very interested in using this package, and have started incorporating it into some tooling. The scipy version is currently pinned to 1.7.3, which only supports up to Python 3.10. I was wondering if you think there are any barriers to actually bumping that up to a more recent version of scipy, or whether you have plans to update for support of 3.11, or if you'd be open to a PR to make that happen. Thanks!

API documentation issues - JOSS review

This issue outlines some minor API documentation issues as part of the review for JOSS:

It would be useful if the API documentation specified the full path to the function. E.g. instead of documenting "con1way" as appears here, the full path to the function could be given: e.g. hypothesize.utilities.con1way. Take the API reference of sklearn as an example.

This way, t would be easier for the user to import functions.

Software paper issues - JOSS review

This issue outlines some minor issues with the software paper as part of the review for JOSS.

1. Since the JOSS guidelines require "a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience" it would be nice to have a short definition of what a robust statistical method is before mentioning the advantages it has over other approaches.

2. When mentioning the R library created by Wilcox, I believe the following citation should be added: https://link.springer.com/article/10.3758/s13428-019-01246-w

3. When describing the target audience of Hypothesize, I would add that this toolbox does assume knowledge on how to perform robust analyses. The description of the methods in the API or manipulating the method's parameters assumes familiarity with concepts and acronyms used in the field (e.g. FWE, MOM). This is by no means a criticism of the toolbox per se, it is just a suggestion to better clarify who are the target users of Hypothesize.

Provide documentation and dependencies for running tests

The contributing documentation describes how tests are structured and tools used, but doesn't actually say how to run the tests. I think adding a sentence or two describing how to run the tests would be useful. Perhaps something like:

Following installation, to run the test suite first navigate to the tests directory then use the pytest command.

Additionally, in the setup.py it would be useful to add tests_require=['pytest'] for completeness.

p-value with value 0

Increasing precision doesn't return the exact number.

Use NumPy generators instead of global state

Hi, great library 👏

I have a suggestion for the seeding part. In NEP19, we recommend now to move away from the global state np.random.seed. Instead, the canonical way of using RNG is to use a generator with: rng = np.random.default_rng(seed_value). (Would be ok since you require a recent version of NumPy.)

This is especially important since in SciPy for e.g. we are not relying anymore on the global state but ask people to use the parameter random_state (sometimes called seed or rng depending on the function, sorry about the inconsistency, we are trying to resolve that slowly...).

JOSS paper suggestions

Hi @Alcampopiano - thanks for your work in creating hypothesize! I think it's a really nice piece of software and a good contribution to JOSS. However I think the paper in its current form could be improved to make it more descriptive and stand on it's own a bit better. These are my suggestions:

1. In the text of the paper you list a couple of other packages for robust statistics, and say that hypothesize implements more methods. Could you elaborate on what was missing from these other packages and/or what problems hypothesize can solve that these others cannot?

2. I think your documentation is really great, and you could bring some of this information into the paper as well. Namely, I think including the package structure and list of estimators from the overview page would be really helpful for making it clear what is in hypothesize.

function roadmap for version 1.0

A small subset of Wilcox's functions that should be included in the first major release.

utilities

• con1way and con2way for creating linear contrasts

two independent groups

• yuenbt bootstrapped-t CI for ut1 - ut2
• pb2gen percentile bootstrap CI for difference between any estimators

two dependent groups

• ydbt bootstrapped-t CI for ut1 - ut2
• bootdpci percentile bootstrap method any estimator; can set options for using difference scores or measures of location based on the marginal distributions
• loc2dif and l2drmci difference between estimators using all combinations of difference scores and percentile bootstrap, respectively

one-way methods based on multiple comparisons for independent groups

• linconb test linear contrasts using bootstrap-t method
• tmcppb rom/hoch/ben-type methods using percentile bootstrap and trimmed means

one-way methods based on multiple comparisons for dependent groups

• rmmcppb multiple comparisons for J dependent groups using percentile boot method
• lindepbt bootstrap-t method for multiple comparisons among J dependent groups

within-within designs

• wwmcppb multiple comparisons for main effects and interactions with linear contrasts using percentile boot
• wwmcpbt multiple comparisons for main effects and interactions with linear contrasts using bootstrap-t

mixed designs

• bwmcp (bwamcp, bwbmcp, bwimcp) all main effects and interactions using bootstrap-t
• spmcpa, spmcpb, spmcpi like the above functions but using the percentile boot
• bwmcppb (only for trimmed means?) all main effects and interactions with percentile boot

between-between designs ?

correlations

• pbcor percentage bend correlation
• pball percentage bend correlation for a set of variables
• wincor winsorized correlation
• winall winsorized correlation for a set of variables
• corb test for zero correlation using bootstrapping

unit testing

Write unit tests for top-level functions

trim_mean not defined for 'est'

On Python 3.7 on Windows 10 (Anaconda/Jupyter notebook)

for compare_groups_with_single_factor.bootdpci the only arugment to the est function allowable is trim_mean:

https://alcampopiano.github.io/hypothesize/function_guide/#bootdpci

est: function

Measure of location (currently only trim_mean is supported)

However on

import hypothesize as hypo
hypo.compare_groups_with_single_factor.bootdpci(xdf,est=trim_mean)

NameError: name 'trim_mean' is not defined

If other estimators, such as numpy.mean, are used

hypo.compare_groups_with_single_factor.bootdpci(xdf,est=np.mean)

the following error is thrown:

analysis is being done on difference scores each confidence interval has probability coverage of 1-alpha.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-39-c2264eb089c1> in <module>
      1 # https://alcampopiano.github.io/hypothesize/function_guide/#bootdpci
----> 2 hypo.compare_groups_with_single_factor.bootdpci(xdf,est=np.mean)

~\anaconda3\envs\oepython37\lib\site-packages\hypothesize\compare_groups_with_single_factor\_compare_groups_with_single_factor.py in bootdpci(x, est, nboot, alpha, dif, BA, SR, *args)
   1024 
   1025     results=rmmcppb(x, est, *args, nboot=nboot,alpha=alpha,
-> 1026                    SR=SR, dif=dif, BA=BA)
   1027 
   1028     col_names = ['con_num', 'psihat', 'p_value', 'p_crit', 'ci_lower', 'ci_upper']

~\anaconda3\envs\oepython37\lib\site-packages\hypothesize\compare_groups_with_single_factor\_compare_groups_with_single_factor.py in rmmcppb(x, est, alpha, con, dif, nboot, BA, hoch, SR, seed, *args)
    322 
    323         temp=rmmcppbd(x,est, *args, alpha=alpha,con=con,
--> 324                       nboot=nboot,hoch=True)
    325 
    326         if called_directly:

~\anaconda3\envs\oepython37\lib\site-packages\hypothesize\compare_groups_with_single_factor\_compare_groups_with_single_factor.py in rmmcppbd(x, est, alpha, con, nboot, hoch, seed, *args)
    646 
    647     for ic in range(ncon):
--> 648       output[ic, 1] = tmeans[ic]
    649       output[ic, 0] = ic
    650       output[ic, 2] = test[ic]

IndexError: invalid index to scalar variable.