t-test bootstrapped p-values very different compared to permutation

[WillemSleegers]

t-test bootstrapped p-values very different compared to permutation

[simonpcouch]

Thanks for the issue, @WillemSleegers.

This result is indeed expected; the distributions resulting from those two randomization procedures are quite different. We can visualize those distributions to better understand how those p-values are derived:

library(infer)

set.seed(1)

observed_statistic <- gss %>%
   specify(hours ~ college) %>%
   calculate(stat = "diff in means", order = c("degree", "no degree"))

observed_statistic
#> Response: hours (numeric)
#> Explanatory: college (factor)
#> # A tibble: 1 × 1
#>    stat
#>   <dbl>
#> 1  1.54

boot_dist <- gss %>%
   specify(hours ~ college) %>%
   hypothesize(null = "independence") %>%
   generate(reps = 1000, type = "bootstrap") %>%
   calculate(stat = "diff in means", order = c("degree", "no degree"))

permute_dist <- gss %>%
   specify(hours ~ college) %>%
   hypothesize(null = "independence") %>%
   generate(reps = 1000, type = "permute") %>%
   calculate(stat = "diff in means", order = c("degree", "no degree"))

visualize(boot_dist) + shade_p_value(observed_statistic, direction = "two-sided")

visualize(permute_dist) + shade_p_value(observed_statistic, direction = "two-sided")

^{Created on 2024-01-08 with reprex v2.0.2}

The distribution derived from permutation is centered at 0 since any association between hours and college has been disrupted via permuting one of the variables. The bootstrap distribution is centered at the observed statistic since whole rows are bootstrapped together, preserving any association present in the inputted data.

[WillemSleegers]

Thanks for responding @simonpcouch!

I now realize it works as intended (so, sorry for making the issue!), although it's too bad it doesn't work to obtain a similar test compared to that of permute. If I understand it correctly, the bootstrap version could be obtained by sampling from each group after subtracting the observed mean of each group from each group's data point, although I suppose this might not work with how infer works under the hood.

Either way, feel free to close the issue!

[simonpcouch]

You're welcome!

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.