Possibility of calculating the nonparametric permutation test and checking one-sided hypotheses (alternative greater and less)

[otavares93]

Possibility of calculating the nonparametric permutation test and checking one-sided hypotheses (alternative greater and less)

[qbarthelemy]

If you have the same number of observations for the 2 samples you are testing (data augmentation versus no data augmentation), you can use a paired Wilcoxon test. Otherwise, you must use a Mann-Whitney test.

A low value of T (resp. U) statistic for Wilcoxon (resp. Mann-Whitney) two-sample test is required for statistical significance.
This is why we compute the min on statistics of multiple variables (in your case, you have only one variable) to build the null distribution, and why we apply a left-sided test to it.

This statistical processing takes place even if you seek to maximize the value of your observations (precision, specificity or AUC).

[qbarthelemy]

@otavares93, if it is ok for you, can I let you close the issue?

[qbarthelemy]

@otavares93, can you describe your experimental setup more specifically (number of samples, number of variables, null hypothesis, alternative hypothesis, etc)?

[otavares93]

@qbarthelemy, sure ! I have two classifier results (i.e. sensitivity, specificity, auc). The two classifiers have the same architecture, but one of them uses raw data as input and the second uses the raw data plus a data augmentation strategy. I want to test the hypothesis of whether the data augmentation strategy plays a role in the classifier performance. Then, my null hypothesis is H0: the classifier's results come from the same distribution. H1: the classifier's results come from different distributions. I have two experiments, one gives me 90 stats calculated in the test set for each classifier (k-fold with 10 folds - train, val, test) and I have 10 stats within the second (only the best models selected after the training process - train, val - are evaluated on the test set). The idea to use the one-sided test is because I seek to check the evidence of one of the classifiers points to better results once compared to the other. I calculated the Wilcoxon test using the Scipy module but without the permutation approach. Therefore, I would like to apply the tests mentioned above on my example using the permutation approach implemented in your package.

[otavares93]

Dear @qbarthelemy, yes, it's pretty ok. Thank you for your enlightening answer. best regards !