Comments (5)
If you have the same number of observations for the 2 samples you are testing (data augmentation versus no data augmentation), you can use a paired Wilcoxon test. Otherwise, you must use a Mann-Whitney test.
A low value of T (resp. U) statistic for Wilcoxon (resp. Mann-Whitney) two-sample test is required for statistical significance.
This is why we compute the min on statistics of multiple variables (in your case, you have only one variable) to build the null distribution, and why we apply a left-sided test to it.
This statistical processing takes place even if you seek to maximize the value of your observations (precision, specificity or AUC).
from pypermut.
@otavares93, if it is ok for you, can I let you close the issue?
from pypermut.
@otavares93, can you describe your experimental setup more specifically (number of samples, number of variables, null hypothesis, alternative hypothesis, etc)?
from pypermut.
@qbarthelemy, sure ! I have two classifier results (i.e. sensitivity, specificity, auc). The two classifiers have the same architecture, but one of them uses raw data as input and the second uses the raw data plus a data augmentation strategy. I want to test the hypothesis of whether the data augmentation strategy plays a role in the classifier performance. Then, my null hypothesis is H0: the classifier's results come from the same distribution. H1: the classifier's results come from different distributions. I have two experiments, one gives me 90 stats calculated in the test set for each classifier (k-fold with 10 folds - train, val, test) and I have 10 stats within the second (only the best models selected after the training process - train, val - are evaluated on the test set). The idea to use the one-sided test is because I seek to check the evidence of one of the classifiers points to better results once compared to the other. I calculated the Wilcoxon test using the Scipy module but without the permutation approach. Therefore, I would like to apply the tests mentioned above on my example using the permutation approach implemented in your package.
from pypermut.
Dear @qbarthelemy, yes, it's pretty ok. Thank you for your enlightening answer. best regards !
from pypermut.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pypermut.