Inquiry Regarding Confidence Interval Behavior in createSensitivityResults_relativeMagnitudes() Function

[asheshrambachan]

Inquiry Regarding Confidence Interval Behavior in createSensitivityResults_relativeMagnitudes() Function

[jonathandroth]

I would guess it's a numerical precision issue (HonestDiD inverts tests over a finite grid of points and then collects them to form the CI). Can you try re-running with fixed values of the upper and lower bounds for the grid (parameters grid.ub and grid.lb) and tell me if that fixes it?

…

On Mon, Jan 8, 2024 at 6:10 PM Ziyi Liu ***@***.***> wrote: I hope this message finds you well. Firstly, I would like to express my appreciation for the development of your package, which I've found to be extremely convenient and useful in my work. However, I am encountering a puzzling issue while utilizing the createSensitivityResults_relativeMagnitudes() command, specifically in relation to the generation of confidence intervals under varying values of M_bar. When I adjust M_bar to larger values, the resulting confidence intervals become narrower, which appears somewhat counter-intuitive. To illustrate, consider the following output: honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe, sigma = vcov.twfe, numPrePeriods = 12, numPostPeriods = 1, l_vec = c(1), Mbarvec = c(0,0.25,0.5,0.75,1), gridPoints = 3000) honest.twfe A tibble: 5 x 5 lb ub method Delta Mbar 1 0.0544 0.177 C-LF DeltaRM 0 2 0.0565 0.175 C-LF DeltaRM 0.25 3 0.0516 0.178 C-LF DeltaRM 0.5 4 0.0359 0.193 C-LF DeltaRM 0.75 5 0.0187 0.209 C-LF DeltaRM 1 Notably, the confidence interval for M_bar = 0.25 is narrower than for M_bar = 0. Additionally, this interval contrasts with the one generated using constructOriginalCS(), which is considerably wider: constructOriginalCS(betahat = beta.twfe, sigma = vcov.twfe, numPrePeriods = 12, numPostPeriods = 1, l_vec = c(1)) A tibble: 1 x 4 lb[,1] ub[,1] method Delta 1 0.0786 0.154 Original NA I am seeking some insights or explanations regarding this behavior. Your guidance on understanding these discrepancies would be immensely helpful. Additionally, if necessary, I can provide a working example for further investigation. Thank you for your time and assistance. Best regards, — Reply to this email directly, view it on GitHub <#49>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE6EXFDM6W2V42XKQ42EYA3YNR4FTAVCNFSM6AAAAABBSGLMRSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA3TCMZWHEZTGNI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[lzy318]

Thanks for your prompt response, I re-run the codes with fixed grid but the problem persists. Here is my code:

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe,
                                                           sigma = vcov.twfe,
                                                           numPrePeriods = 12,
                                                           numPostPeriods = 1,
                                                           l_vec = c(1),
                                                           Mbarvec = c(0,0.25,0.5,0.75,1),
                                                           grid.ub = 0.8,
                                                           grid.lb = -0.8)

      lb    ub method Delta    Mbar
   <dbl> <dbl> <chr>  <chr>   <dbl>
1 0.0553 0.175 C-LF   DeltaRM  0   
2 0.0569 0.174 C-LF   DeltaRM  0.25
3 0.0521 0.177 C-LF   DeltaRM  0.5 
4 0.0360 0.191 C-LF   DeltaRM  0.75
5 0.0200 0.209 C-LF   DeltaRM  1   

Let me know if you need the data to replicate this issue. And I am still confused about the difference between the confidence interval when Mbar equals to zero and the confidence interval given by constructOriginalCS(). They should be similar right? But in my case they are not.

[jonathandroth]

Can you please share a reproducible example with data?

…

On Tue, Jan 9, 2024, 2:33 PM Ziyi Liu ***@***.***> wrote: Thanks for your prompt response, I re-run the codes with fixed grid but the problem persists. Here is my code: honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe, sigma = vcov.twfe, numPrePeriods = 12, numPostPeriods = 1, l_vec = c(1), Mbarvec = c(0,0.25,0.5,0.75,1), grid.ub = 0.8, grid.lb = -0.8) lb ub method Delta Mbar <dbl> <dbl> <chr> <chr> <dbl> 1 0.0553 0.175 C-LF DeltaRM 0 2 0.0569 0.174 C-LF DeltaRM 0.25 3 0.0521 0.177 C-LF DeltaRM 0.5 4 0.0360 0.191 C-LF DeltaRM 0.75 5 0.0200 0.209 C-LF DeltaRM 1 Let me know if you need the data to replicate this issue. And I am still confused about the difference between the confidence interval when Mbar equals to zero and the confidence interval given by constructOriginalCS(). They should be similar right? But in my case they are not. — Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE6EXFBR252Z55I5DXTX3VTYNWLRHAVCNFSM6AAAAABBSGLMRSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBTGY2TSNZXGA> . You are receiving this because you commented.Message ID: ***@***.***>

[lzy318]

Thanks for the quick reply, this is the link to the data.

https://drive.google.com/file/d/14IoDrnCulGmUfCyBJkqmNp-iObSIgw_3/view?usp=sharing

Here is the code for the event study.

`rm(list = ls())
library(HonestDiD)
library(fixest)
load("example.RData")
twfe.est <- feols(general_sharetotal_A_all ~ i(Time_to_Treatment, treat, ref = -1) + cand_H_all + cand_B_all| district_final + cycle,
data = data.twfe, cluster = "district_final")

beta.twfe <- twfe.est$coeftable[c(4:16),1]
vcov.twfe <- summary(twfe.est)$cov.scaled[c(4:16),c(4:16)]

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe,
sigma = vcov.twfe,
numPrePeriods = 12,
numPostPeriods = 1,
l_vec = c(1),
Mbarvec = c(0,0.25,0.5,0.75,1),
grid.ub = 0.8,
grid.lb = -0.8)
honest.twfe
original.twfe <- constructOriginalCS(betahat = beta.twfe,
sigma = vcov.twfe,
numPrePeriods = 12,
numPostPeriods = 1,
l_vec = c(1))
original.twfe`

[jonathandroth]

Hmm, that is weird. I am able to replicate your strange result. However, if I include the second post-treatment coefficient and then tell it I'm only interested in the first one (l_vec = c(1,0)), then I get the expected result that the CI for M=0 is nearly identical to the original CI, and the CIs expand as M increases.

library(fixest)
load("~/Downloads/example.RData")
twfe.est <- feols(general_sharetotal_A_all ~ i(Time_to_Treatment, treat, ref = -1) + cand_H_all + cand_B_all| district_final + cycle,
                  data = data.twfe, cluster = "district_final")

beta.twfe <- twfe.est$coeftable[c(4:17),1]
vcov.twfe <- summary(twfe.est)$cov.scaled[c(4:17),c(4:17)]

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe,
                                                           sigma = vcov.twfe,
                                                           numPrePeriods = 12,
                                                           numPostPeriods = 2,
                                                           l_vec = c(1,0),
                                                           Mbarvec = c(0,0.25,0.5,0.75,1),
                                                           grid.ub = 0.8,
                                                           grid.lb = -0.8)

IIRC, HonestDiD has a special subroutine for the case without nuisance parameters (i.e. only one post-treatment period), and I'm wondering if there is a bug there. @mcaceresb when you have some RA time, could you look into this please?

CCing @asheshrambachan since I think you wrote this code initially and may have a better idea where the bug is.

[mcaceresb]

@jonathandroth The critical values are different.

1. The least-favorable critical value is larger here when using the "no-nuisance" parameter logic.

   • This call does not pass X_T or rowsForARP, contrast with this call.
   • As a result, they use different parts of this if-else block and give different critical values.
   • The no-nuisance logic and the nuisance logic have the same lhs but different rhs as a result.

2. If the least-favorable CV is not rejected, the next critical value is obtained from the generalized normal, which is also different.

   • This no-nuisance call uses the values returned from .VLoVUpFN for the lower and upper bounds, whereas this nuisance call uses the values from .lp_dual_fn; I'm not quite sure why but they end up being rather different, but they do.