asheshrambachan/honestdid

Robust inference in difference-in-differences and event study designs

honestdid's Issues

Problems with computeConditionalCS_DeltaSD and createSensitivityResults

Hello,

I have problems replicating the results when I run the code that appears in your vignette. When I run the command CreateSensitivityResults I get the error "Error in { : task 1 failed - "Incomparable dimensions" and when I run the command ComputeConditionalCS_DeltaSD I have the error message "mul_dims_promote(lh_dim, rh_dim): incomparable dimensions". The error messages refer to incomparable dimensions not to "non conformable arguments" (that I get if I choose randomly the number of pre and post periods).

When I googled the error message, I found a link to a R/utilities page describing some code for CVXR Disciplined Convex Optimization.

I re-installed the package and tried on several computers. All the packages that I was asked to update when installing HonestDiD re up to date. I use Windows 10 but had the same problems on a Mac (Mojave). I run R 3.6.2 version.

Tell me if you need any other information.

Handling of universal base period

Hi there,

maybe the question is trivial but I couldn't really figure it out by reading the paper and README of the repo: How would I specify numPrePeriods and numPostPeriods in the call to createSensitivityResults_relativeMagnitudes when my base period is some pre-treatment period other the canoncial "-1"?

Let's say my event study estimates range from -8 to 8, with 0 being the first treatment period and having defined -4 as the reference period to which the estimates are normalized (due to anticipation effects). Would I then define numPrePeriods = 4 (-8,-7,-6,-5) or numPrePeriods = 7 (-8,-7,-6,-5,-3,-2,-1). I guess my issue here is that in the second case pre-treatment trends would be drawn from -5 to -3 right?

Grateful for any hints!

Dependencies

This is an excellent package and really helpful readme. Easy to follow and the results replicated quickly on my system.

When following the readme I was asked to install tidyverse and Rglpk but these are not mentioned in the readme as being needed.

Hitting the upper bound?

Hello! Thank you for the package and for your help!

I'm using the did package and conducting sensitivity analysis with the honest_did auxiliary function.

At some level of $\bar{M}$, the confidence intervals stop increasing, perhaps because they've reached the plot's upper/lower bound.

Here is the code used to produce the result, with period 7 as an example:

att2 <- att_gt(yname = "vote",
               tname = "time.id",
               idname = "ID",
               gname = "firstrep",
               xformla = ~age + female,
               data = panel.df,
               base_period = "universal",
               control_group = "notyettreated") 

es2 <- aggte(att2, type = "dynamic", na.rm = T, min_e = -12, max_e = 12)

sensitivityresults7 <- honest_did(es = es2, 
                   e = 7,
                   type="relative_magnitude")

sensitivityresults7$robust_ci <- sensitivityresults7$robust_ci[-1,]

createSensitivityPlot_relativeMagnitudes(sensitivityresults7$robust_ci,
                                                    sensitivityresults7$orig_ci)

The output is:

The data can be found at: https://drive.google.com/drive/folders/1N5HEWIIZSX_EgKzrmX9R9hzcn5fE5zyq?usp=sharing

Do you have a suggestion to expand the bounds using this approach?

Thank you!!

Installation troubles

Greetings. I am having trouble installing HonestDID using the instructions provided in the vignette. This is odd, since I had no issues installing last summer on my old laptop. I'm using a Windows OS with the most recent version of R Studio.

Problems installing the package

Hello,
I tried to download the package following the steps you highlighted but get the following error message

installation of package ‘.../Local/Temp/RtmpOeI1SA/file418039153388/HonestDiD_0.1.0.tar.gz’ had non-zero exit status

I tried to google the message and find a solution but did not find one. Would you have tips?
Thanks in advance,

Pascal

A problem when i run command HonestDiD::createSensitivityResults_relativeMagnitudes

Thank you for creating this tool. It sure looks interesting and is something I would like to use for an upcoming project!
I need your help though

An error occurred when I run the R command from the example:

delta_rm_results <-
HonestDiD::createSensitivityResults_relativeMagnitudes(
betahat = betahat, #coefficients
sigma = sigma, #covariance matrix
numPrePeriods = 5, #num. of pre-treatment coefs
numPostPeriods = 2, #num. of post-treatment coefs
Mbarvec = seq(0.5,2,by=0.5) #values of Mbar
)

Then the console starts a procedure that seems to never stop. And when I stop it, it tells me there are more than 50 warning messages. Taking a look at these with warnings() two warnings are repeated over and over again. These:
Warning messages:
1: In seq_len(length.out - 2L) * (del/n1) :
Recycling array of length 1 in vector-array arithmetic is deprecated.
Use c() or as.vector() instead.

2: In from + seq_len(length.out - 2L) * (del/n1) :
Recycling array of length 1 in array-vector arithmetic is deprecated.
Use c() or as.vector() instead

3: In seq_len(length.out - 2L) * (del/n1) :
Recycling array of length 1 in vector-array arithmetic is deprecated.
Use c() or as.vector() instead.

4: In from + seq_len(length.out - 2L) * (del/n1) :
Recycling array of length 1 in array-vector arithmetic is deprecated.
Use c() or as.vector() instead.

And the rest are all the same.

How should I deal with this?

Error in vignette

Thanks for creating this package.

Trying to follow along with the vignette, using a clean R installation and installing using devtools.

Found that createSensitivityResults hangs

DeltaSD_RobustResults = createSensitivityResults(betahat = LWdata_EventStudy$betahat,
sigma = LWdata_EventStudy$sigma,
numPrePeriods = numPrePeriods,
numPostPeriods = numPostPeriods,
l_vec = l_vec,
Mvec = seq(from = 0, to = 0.04, by = 0.005))

Observed the below warning. Any suggestions?

Recycling array of length 1 in vector-array arithmetic is deprecated.
Use c() or as.vector() instead.

Thanks in advance!
Russ

Error when running createSensitivityResults_relativeMagnitudes - Caused by error in `svd()`: ! a dimension is zero"

Hi,

thanks a lot for your package! I am having some trouble running HonestDiD::createSensitivityResults_relativeMagnitudes() in my data. When I try running it I get the following error:

Error in { : task 1 failed - "ℹ In index: 599. Caused by error in svd(): ! a dimension is zero"

I am having a hard time understanding what this error means and how to inspect for issues in my code/regression. If you could help me with some guidance on what to do that would be great! Below is an example code of what I am running (let me know if you need a working example to dig into this issue):

parallel_trends_levels_class_year <- fixest::feols(deforestation ~ i(year, class_dummy, ref = 2018)| class + year, cluster = "city_id", reg_data %>% filter(year>2007))

betahat <- summary(parallel_trends_levels_class_year)$coefficients

sigma <- summary(parallel_trends_levels_class_year)$cov.scaled

parallel_trends_levels_class_year_sensitivity <- HonestDiD::createSensitivityResults_relativeMagnitudes( betahat = betahat, sigma = sigma, numPrePeriods = 10, numPostPeriods = 4, Mbarvec = 0.5 )

A problem when i run command HonestDiD::createSensitivityResults_relativeMagnitudes

Thank you for creating this tool.
I need your help.
An error occurred when I run the R command from the example:
(
**delta_rm_results <-
HonestDiD::createSensitivityResults_relativeMagnitudes(
betahat = betahat, #coefficients
sigma = sigma, #covariance matrix
numPrePeriods = 5, #num. of pre-treatment coefs
numPostPeriods = 2, #num. of post-treatment coefs
Mbarvec = seq(0.5,2,by=0.5) #values of Mbar
)

delta_rm_results**
)
then console show the below error
error in no package with the name 'Rglpk' exists.
how to deal with this problem, thanks!

Add documentation on how to import a regression from STata

Example using the DRDID package?

Hello HonestDiD Team,

Do you know if there is an example implementation of this approach that uses the DRDiD method? I see the version for CS estimator, but if we have non-staggered data and are using the DRDiD package, is there an existing implementation for that within HonestDID?

And if not, how hard would it be to do this?

Thanks!

Best,
Josh

Apply honestdid on did2s results

Dear authors, I'm trying to apply the output of did2s to honestdid. But there are errors so I wonder if you have any ideas about it? Thank you very much!!

method <- NULL
Mvec <- NULL
Mbarvec <- NULL
monotonicityDirection <- NULL
biasDirection <- NULL
alpha <- 0.05
parallel <- FALSE

robust_ci <- createSensitivityResults(
    betahat = beta,
    sigma = V,
    numPrePeriods = npre,
    numPostPeriods = npost,
    method = method,
    l_vec = baseVec1,
    monotonicityDirection = monotonicityDirection,
    biasDirection = biasDirection,
    alpha = alpha,
    parallel = parallel
  )

I got an error message Error in { : task 1 failed - "attempt to apply non-function". I wonder if you have any ideas? Thank you very much!

Add explicit calls to buit-in functions

Following up on #21 and #22, per this comment, I should make all calls to built-in functions explicit as well (i.e. base::sqrt instead of sqrt, etc.)

Question about dimensions of dimensions of numPost/PrePeriods

Hello,

My setting is the following, I observe regions i for three periods, -1, 0 and 1. Treatment happens in certain regions between 0 and 1 (very much like your motivating example). I estimate the following equation, D is a dummy for treated region

y_{i,t} = \alpha + \lambda_{t} + \omega_{i} + \beta_{-1}D{year=-1} + \beta_{1}*D{year=1} + time varying controls + \eps_{i,t}

where \beta_{-1} is the placebo and \beta_{0} is the omitted/normalized coefficient, this is your equation (1).

When I create the list (as you did with LWdata_EventSudy), betahat and sigma are of dimension 2 and numPre and Postperiods are both 1. This is how I understand you dealt with the omitted period. However when I do that, I receive the following error message (when I run createSensitivityResults)

Error in [<-(*tmp*, r, r:(r + 2), value = c(1, -2, 1)) :
subscript out of bounds

The code does run when I add zeros for \beta_{0} in both the vector of coefficients and the covariance matrix. The dimension of everything goes from two to three (and I change numPrePeriods to 2) but it seems an odd manipulation.

My questions are:

• In my case; what should be the dimensions of bethat, sigma and numPre/PostPeriods?
• How odd is me manually adding zeros?
• Is there something I got wrong about the way you dealt with the missing coefficient in creating LWdata_EventStudy?

Thanks again,
Best,
Pascal

Package dependencies and package conflicts

You have already partially addressed this in a previous issue, but the way the current package dependencies are set up can cause conflicts depending on the other packages loaded into the environment, or even the order that those packages are loaded into the environment. It is also making it difficult to install on some of our cluster computing environments.

There are a couple small changes that could be made to resolve 90% of these issues:

1. Move most of the packages listed in the DESCRIPTION file from Depends: to Imports:. Depends forces all those packages to be loaded into the environment at the same time as your package, which is usually not necessary. It's often sufficient to just have them available via explicit call (see next suggestion).

2. Make all the calls to external functions explicit, package::function() rather than just function(). This will be necessary for packages listed in Imports, as they will not be natively loaded into the environment unlike the Depends packages.

3. Replace tidyverse in your dependencies list with the component packages. You currently include tidyverse in your Depends list, but tidyverse is a wrapper around multiple packages to allow for quicker loading. It is currently unclear which of the underlying packages nor which versions are needed.

Query about CIs under different values of Mbar

Hello,

Many thanks in advance if you're able to help at all. I've been playing around with the example data and performing the sensitivity analysis based on relative magnitudes in the staggered DiD case (using the function created by Pedro Sant'Anna) and have a query regarding the resulting confidence intervals.

When I use a smaller increment between values of Mbar (e.g. Mbarvec = seq(from = 0.1, to = 1, by = 0.1)), some of the confidence intervals are identical for adjacent Mbar values. Would this just be a consequence of the precision of the code?

Also, for very small values of Mbar (e.g. less than 0.3) the resulting confidence intervals are actually narrower than the original CI which does not seem intuitive to me. I wonder if this due to being close to Mbar=0 which is "not really allowed" (https://github.com/pedrohcgs/CS_RR) and thus we shouldn't use very small values of Mbar?

Both of these findings also occurred when I implemented the methods on my own data.

Thanks a lot!

What's the convention for indices that are not specified?

In the paper you index beta and delta with negative indices for pre-treatment periods, and with positive indices for post-treatment periods. So, when constructing Delta^{SD}(M) for example, which includes expressions with delta_{t-1}, and delta_{t+1}, what is the convention for indices that are not defined? ie, how do I interpret expressions like delta_0, delta_(upperT+1), or delta_{lowerT-1}?

Graphing for estimator from Callaway and Sant’Anna (2020)

Warning in install.packages: package ‘HonestDiD’ is not available for this version of R

Hello,
Thanks for creating this DID package, a nice paper as well.

I am trying to install your package and there is a warning saying "package ‘HonestDiD’ is not available for this version of R".
Which version of R can run your package?
Please see the photo for more info.

Honest_did function not working

Hello,
I am trying to implement the honest_did paralell trends test to my results, but I get an error that the honest_did no longer works. Is the comment discontinued?

`nightl1 <- nightl %>%
filter(year >1992, d50_EXT == 0 & d50_ENLDD == 0 & d50_ENLN3 == 0 & d50_EFTA == 0 & d50_EXTHR ==0,
d100_ENLE8 == 1 | d100_ENLE2 == 1 & ENLE8 > 50 | d100_ENLHR ==1 & ENLE8 > 50,
d2530_ENLE8 == 0 & d3035_ENLE8 == 0 & d3540_ENLE8 == 0 & d4045_ENLE8 == 0 & d4550_ENLE8 == 0,
d2530_ENLE2 == 0 & d3035_ENLE2 == 0 & d3540_ENLE2 == 0 & d4045_ENLE2 == 0 & d4550_ENLE2 == 0,
d2530_ENLHR == 0 & d3035_ENLHR == 0 & d3540_ENLHR == 0 & d4045_ENLHR == 0 & d4550_ENLHR == 0,
CNTR_CODE %in% c("EE", "LV", "LT", "PL", "CZ", "SK", "HU", "SI", "BG", "HR", "RO", "AT", "DE", "DD", "IT", "EL")) %>%
mutate(treatment = ifelse(d25_ENLE8 == 1 & nearest_group == "ENLE8", 2004,
ifelse(d25_ENLE2 == 1 & nearest_group == "ENLE2" , 2007,
ifelse(d25_ENLHR == 1 & nearest_group == "ENLHR" , 2013, 0))))
a <- nightl1 %>%
group_by(treatment) %>%
tally()
knitr::kable(a)

nightl1 %<>%
group_by(id) %>%
mutate(id = cur_group_id()) %>%
ungroup()

nightl1 <- as.data.frame(nightl1)

res<- att_gt(yname = "Delta_NL", tname = "year", idname = "id", gname = "treatment", data = nightl1, control_group = "notyettreated", base_period = "universal", clustervars = "nuts3")

es <- did::aggte(res, type = "dynamic",
min_e = -5, max_e = 5)

sensitivity_results <-
honest_did(es,
e=0,
type="relative_magnitude",
Mbarvec=seq(from = 0.5, to = 2, by = 0.5))

`

Error in honest_did(es, e = 0, type = "relative_magnitude", Mbarvec = seq(from = 0.5, :
could not find function "honest_did"

Error in { : task 1 failed - "non-conformable arguments"

Hello!

First, thank you for your great work and the package. I am experiencing some issues pertaining to non-conformable arguments.

I am estimating a fixed-effects regression with entity and time fixed effects (with plm, using a within estimation, individual fixed effects and manually added time dummies). Subsequently, I adjust standard errors and cluster by group. I then wanted to employ the tests that you have described in the vignette.

Replicating the vignette works fine for me. However, when I try to apply the same steps to my own data, estimation fails. The structure of my data appears to be very similar, albeit a difference in the number of pre periods and more coefficients. Could the failure have something to do with the structure of sigma? I have posted the console output below. Hope this is an appropriate place to flag my issue.

> HKLO_numPrePeriods
[1] 5
> HKLO_numPostPeriods
[1] 4
> HKLO_l_vec
     [,1]
[1,]    1
[2,]    0
[3,]    0
[4,]    0
> beta_1
 [1]  0.025639471  0.012862650  0.031613393  1.214880752  0.043718345  0.050889448  0.020796999 -0.006101869  0.077520292
[10]  0.013092666  0.012840537 -0.013910122
> sigma_1
               [,1]          [,2]          [,3]          [,4]          [,5]          [,6]          [,7]          [,8]
 [1,]  5.960917e-05 -3.948848e-05 -1.861408e-05  0.0001701266 -1.272809e-06 -2.390593e-06 -2.523283e-06 -5.074482e-06
 [2,] -3.948848e-05  2.676426e-04  5.116043e-06  0.0001324583 -3.985026e-06 -8.084589e-06 -1.329489e-05 -1.825596e-05
 [3,] -1.861408e-05  5.116043e-06  1.492741e-04 -0.0005439702 -1.200232e-05 -2.171110e-05 -2.940021e-05 -1.481694e-05
 [4,]  1.701266e-04  1.324583e-04 -5.439702e-04  0.0193808492 -3.796138e-04 -7.724937e-04 -1.180764e-03 -1.706985e-03
 [5,] -1.272809e-06 -3.985026e-06 -1.200232e-05 -0.0003796138  1.317267e-05  2.476562e-05  3.611966e-05  4.652359e-05
 [6,] -2.390593e-06 -8.084589e-06 -2.171110e-05 -0.0007724937  2.476562e-05  4.916163e-05  7.186124e-05  9.315773e-05
 [7,] -2.523283e-06 -1.329489e-05 -2.940021e-05 -0.0011807636  3.611966e-05  7.186124e-05  1.074581e-04  1.399805e-04
 [8,] -5.074482e-06 -1.825596e-05 -1.481694e-05 -0.0017069845  4.652359e-05  9.315773e-05  1.399805e-04  1.879543e-04
 [9,] -2.350533e-05 -1.105146e-05 -1.147658e-05 -0.0021857571  5.920348e-05  1.177208e-04  1.751695e-04  2.352674e-04
[10,] -2.425382e-05 -1.738534e-05 -1.354381e-05 -0.0026442084  7.036862e-05  1.403867e-04  2.100267e-04  2.826409e-04
[11,] -2.453522e-05 -2.176817e-05 -1.089759e-05 -0.0030996508  8.141667e-05  1.623130e-04  2.427686e-04  3.278943e-04
[12,] -2.663931e-05 -2.742002e-05 -7.637740e-06 -0.0035650541  9.220028e-05  1.841173e-04  2.758906e-04  3.736986e-04
               [,9]         [,10]         [,11]         [,12]
 [1,] -2.350533e-05 -2.425382e-05 -2.453522e-05 -2.663931e-05
 [2,] -1.105146e-05 -1.738534e-05 -2.176817e-05 -2.742002e-05
 [3,] -1.147658e-05 -1.354381e-05 -1.089759e-05 -7.637740e-06
 [4,] -2.185757e-03 -2.644208e-03 -3.099651e-03 -3.565054e-03
 [5,]  5.920348e-05  7.036862e-05  8.141667e-05  9.220028e-05
 [6,]  1.177208e-04  1.403867e-04  1.623130e-04  1.841173e-04
 [7,]  1.751695e-04  2.100267e-04  2.427686e-04  2.758906e-04
 [8,]  2.352674e-04  2.826409e-04  3.278943e-04  3.736986e-04
 [9,]  3.046934e-04  3.627664e-04  4.205758e-04  4.782563e-04
[10,]  3.627664e-04  4.357224e-04  5.045653e-04  5.748301e-04
[11,]  4.205758e-04  5.045653e-04  5.872913e-04  6.690070e-04
[12,]  4.782563e-04  5.748301e-04  6.690070e-04  7.644673e-04
> HKLO_DeltaSDRM_RobustResults = createSensitivityResults_relativeMagnitudes(betahat = beta_1, 
+                                                                            sigma = sigma_1,
+                                                                            bound = "deviation from linear trend",
+                                                                            numPrePeriods = HKLO_numPrePeriods, 
+                                                                            numPostPeriods = HKLO_numPostPeriods,
+                                                                            l_vec = HKLO_l_vec, 
+                                                                            Mbarvec = seq(from = 0, to = 2, by = 0.5), 
+                                                                            gridPoints = 100, grid.lb = -1, grid.ub = 1)
Error in { : task 1 failed - "non-conformable arguments"

Use of alpha in construction of original confidence interval

Quick question... why does the following code for reproducing original confidence intervals use qnorm(1 - alpha) instead of qnorm(1 - alpha/2)?

HonestDiD/R/sensitivityresults.R

Lines 388 to 402 in c926b01

	constructOriginalCS <- function(betahat, sigma,
	numPrePeriods, numPostPeriods,
	l_vec = .basisVector(index = 1, size = numPostPeriods),
	alpha = 0.05) {
	stdError = sqrt(t(l_vec) %*% sigma[(numPrePeriods+1):(numPrePeriods+numPostPeriods), (numPrePeriods+1):(numPrePeriods+numPostPeriods)] %*% l_vec)
	lb = t(l_vec) %*% betahat[(numPrePeriods+1):(numPrePeriods+numPostPeriods)] - qnorm(1-alpha)*stdError
	ub = t(l_vec) %*% betahat[(numPrePeriods+1):(numPrePeriods+numPostPeriods)] + qnorm(1-alpha)*stdError
	return(tibble(
	lb = lb,
	ub = ub,
	method = "Original",
	Delta = NA,
	M = 0
	))
	}

Error when running sensitivity analysis with honest did

Hello there,

I am trying to apply the honestdid function to my dataset. However, these is an error: Error in base::matrix(0, nrow = size, ncol = 1): invalid 'nrow' value (< 0)". I showed detail of the error and it said "nrow = -5".
I tried to extract each metric of the result, such as beta, sigma (V), but I can't found where the error came from.

Any hints are valuable to me!

> **#compute event study**
> cs_nevertreat <- did::att_gt(yname = "xxx",
+                           tname = "year",
+                           idname = "id", 
+                           gname = "y1", 
+                           data = df,
+                           control_group = "nevertreated",
+                           base_period = "universal")
> cs_es_nevertreat<- aggte(cs_nevertreat, type = "dynamic",
+                      min_e = -10, max_e = 8)
>**#sensitivity analysis**
> sensitivity_results <-
+   honest_did(cs_es_nevertreat,
+              e=0,
+              type="relative_magnitude")
**Error in base::matrix(0, nrow = size, ncol = 1) :
invalid 'nrow' value (< 0)**
> **#test code extract sigma, beta, tVec**
> es_nevertreat <- cs_es_nevertreat$inf.function$dynamic.inf.func.e
> n <- nrow(es_never)
Error: object 'es_never' not found
> n <- nrow(es_nevertreat)
> n
[1] 707
> V <- t(es_nevertreat) %*% es_nevertreat / n / n
> nrow(V)
[1] 10
> ncol(V)
[1] 10
> tVec <- cs_es_nevertreat$egt
> beta_es <- cs_es_nevertreat$att.egt
> data.frame(tVec,beta_es)

HonestDiD::createSensitivityPlot_relativeMagnitudes and unused filter

Thank you for both the theoretical contributions and the package!

I am having an issue with the createSensitivityPlot_relativeMagnitudes and createSensitivityPlot functions.

I am able to produce delta_rm_results and originalResults normally. However, when I do the following command:

HonestDiD::createSensitivityPlot_relativeMagnitudes(delta_rm_results, originalResults)

I get the error message:

Error in filter(., Mbar <= maxMbar) : unused argument (Mbar <= maxMbar)

Any guidance on this kind of issue? I was able to use the command without issue for about half a day and then this error appeared. I have tried re-loading the environment and re-installing the package.

Debug functions on special case, numPostPeriods = 1.

The functions computeConditionalCS return errors related to the absence of nuisance parameters in the case of numPostPeriods = 0. See the code below to reproduce the errors:

set.seed(1234567890)
numPrePeriods = 5
numPostPeriods = 1
betahat = c(mvtnorm::rmvnorm(1, mean = c(rep(0, numPrePeriods), rep(0, numPostPeriods))))
sigma = diag(numPrePeriods + numPostPeriods)
l_vec = basisVector(index = 1, size = numPostPeriods)
test_conditionalCS_SD = computeConditionalCS_DeltaSD(betahat = betahat, sigma = sigma,
                                                  numPrePeriods = numPrePeriods, numPostPeriods = numPostPeriods,
                                                  l_vec = l_vec, M = 0, postPeriodMomentsOnly = T)

A simple fix will be to separately implement the underlying ARP functions for the special case without nuisance parameters and call those directly whenever the user specifies numPostPeriods = 1
.

Warning for sigma that is not positive semi-definite

@mcaceresb when you have RA time in the fall, can you add a check for whether sigma is positive semi-definite and if not, throw a warning?

Providing HonestDID with a non-semi-definite Sigma led to some weird error messages, per the email we received from a student.

Inquiry Regarding Confidence Interval Behavior in createSensitivityResults_relativeMagnitudes() Function

I hope this message finds you well. Firstly, I would like to express my appreciation for the development of your package, which I've found to be extremely convenient and useful in my work.

However, I am encountering a puzzling issue while utilizing the createSensitivityResults_relativeMagnitudes() command, specifically in relation to the generation of confidence intervals under varying values of M_bar. When I adjust M_bar to larger values, the resulting confidence intervals become narrower, which appears somewhat counter-intuitive.

To illustrate, consider the following output:

honest.twfe <- createSensitivityResults_relativeMagnitudes(betahat = beta.twfe,
                                                           sigma = vcov.twfe,
                                                           numPrePeriods = 12,
                                                           numPostPeriods = 1,
                                                           l_vec = c(1),
                                                           Mbarvec = c(0,0.25,0.5,0.75,1),
                                                           gridPoints = 3000)
honest.twfe
      lb    ub method Delta    Mbar
   <dbl> <dbl> <chr>  <chr>   <dbl>
1 0.0544 0.177 C-LF   DeltaRM  0   
2 0.0565 0.175 C-LF   DeltaRM  0.25
3 0.0516 0.178 C-LF   DeltaRM  0.5 
4 0.0359 0.193 C-LF   DeltaRM  0.75
5 0.0187 0.209 C-LF   DeltaRM  1   

Notably, the confidence interval for M_bar = 0.25 is narrower than for M_bar = 0. Additionally, this interval contrasts with the one generated using constructOriginalCS(), which is considerably wider:

constructOriginalCS(betahat = beta.twfe,
                    sigma = vcov.twfe,
                    numPrePeriods = 12,
                    numPostPeriods = 1,
                    l_vec = c(1))
  lb[,1] ub[,1] method   Delta
   <dbl>  <dbl> <chr>    <lgl>
1 0.0786  0.154 Original NA   

I am seeking some insights or explanations regarding this behavior. Your guidance on understanding these discrepancies would be immensely helpful. Additionally, if necessary, I can provide a working example for further investigation.

Thank you for your time and assistance.

Best regards,

"incorrect number of dimensions"

Hi,
While trying to create sensitivity results, I encountered an Error in {: task 1 failed - "incorrect number of dimensions."
Please can you take a look? I have not been able to find the root error and resolve it. The code line I am running is below

ghana_DeltaSD_RobustResults = createSensitivityResults(betahat = m1.plm_EventStudy$betahat,

•                                                    sigma = m1.plm_EventStudy$sigma,
  

•                                                    numPrePeriods = ghana_numPrePeriods, 
  

•                                                    numPostPeriods = ghana_numPostPeriods,
  

•                                                    l_vec = ghana_l_vec, 
  

•                                                    Mvec = seq(from = 0, to = 0.04, by = 0.005))
  

Options in `honest_did.AGGTEobj`

In the README, honest_did passes some options but not others:

• Mvec is captured but not passed along. Mbarvec is captured and passed.
• gridPoints is captured and not passed. It's further hard-coded to 100 with relative_magnitude
• grid.ub and grid.lb are both captured and not passed.
• Additional options are allowed via ... but ... is not passed to any function.

It strikes me that you don't need to capture most options and can just pass down .... Then for cases where the syntax is slightly different you can capture both elements but pass both user options (e.g. Mvec and Mbarvec); and for cases where you want a slightly different default you can re-define the option unless the user has specified it. LMK if this is the right way to think about it or if the README is correct and I misunderstood the intent.

Robust CIs narrower than original CIs with large number of pre-treatment periods

Good morning,

I'm not sure if my previous issue sent properly so apologies if this is a duplicate. I'm in a staggered DiD setting with data available in 22 periods before treatment, and then staggered roll-out of treatment over the following 11 periods so e=-34 to e=11 in the event study setup.

When using honestDiD with all pre-periods, a lot of the robust CIs for M=0 (under smoothness restriction) are narrower than the original CIs. The issue remains when increasing the number of gridpoints. However, when I restrict the number of pre-periods in the event study (e.g. min_e = -6) after running Callaway and Sant'Anna's approach, the robust CIs are then wider than the originals. Should the number of pre-periods in the event study be restricted in this case or would the robust CIs using all periods still be valid?

Many thanks,
Kate

constructOriginalCS does not give original confidence intervals + guidance on event studies with controls

Apologies if this is an incorrect interpretation on the desired behavior, but I believe that constructOriginalCS does not quite give the original confidence interval of the original event study passed in. Below is an example using data and calculations from the README:

library(tidyverse)
library(fixest)
library(broom)
library(haven)

df <- haven::read_dta("https://raw.githubusercontent.com/Mixtape-Sessions/Advanced-DID/main/Exercises/Data/ehec_data.dta")

#Keep years before 2016. Drop the 2016 cohort
df_nonstaggered <- df %>% filter(year < 2016 & 
                                   (is.na(yexp2)| yexp2 != 2015) )

#Create a treatment dummy
df_nonstaggered <- df_nonstaggered %>% mutate(D = case_when( yexp2 == 2014 ~ 1,
                                                             T ~ 0)) 

#Run the TWFE spec
twfe_results <- fixest::feols(dins ~ i(year, D, ref = 2013) | stfips + year, 
                              cluster = "stfips",
                              data = df_nonstaggered)


betahat <- summary(twfe_results)$coefficients #save the coefficients
sigma <- summary(twfe_results)$cov.scaled #save the covariance matrix

originalResults <- HonestDiD::constructOriginalCS(betahat = betahat,
                                                  sigma = sigma,
                                                  numPrePeriods = 4,
                                                  numPostPeriods = 3)

Note that I changed the numPrePeriods to 4 andnumPostPeriods to 3 for this example. My understanding is that this will give the original confidence interval (the one specified in the event study) for year 2012. Now when I compare this to the original confidence intervals using broom::tidy(conf.int = T) , I get slightly different results.

twfe_results %>% 
  broom::tidy(conf.int = T) %>% 
  slice(5)

While I do not think this is likely a problem when it comes to analysis, I do think it causes some confusion when trying to understand the package. For instance, I was trying to perform sensitivity analysis on different pre/post-treatment periods (in this case, year 2012), and wanted to create a sensitivity plot for each. However, this discrepancy had (has) me worried that I am misinterpreting how to use the package and incorrectly specifying which period I am testing.

I am curious how to use this package when adding controls to an event study. Here is an example using the data from above, but adding in a fake control column:

df <- haven::read_dta("https://raw.githubusercontent.com/Mixtape-Sessions/Advanced-DID/main/Exercises/Data/ehec_data.dta")

#Keep years before 2016. Drop the 2016 cohort
df_nonstaggered <- df %>% filter(year < 2016 & 
                                   (is.na(yexp2)| yexp2 != 2015) )

#Create a treatment dummy
df_nonstaggered <- df_nonstaggered %>% mutate(D = case_when( yexp2 == 2014 ~ 1,
                                                             T ~ 0)) 

# CREATING A FAKE CONTROL HERE
df_nonstaggered <- df_nonstaggered %>% 
  mutate(control = rnorm(344, 0, 1))

#Run the TWFE spec
twfe_results <- fixest::feols(dins ~ i(year, D, ref = 2013) + control | stfips + year, 
                              cluster = "stfips",
                              data = df_nonstaggered)


betahat <- summary(twfe_results)$coefficients #save the coefficients
sigma <- summary(twfe_results)$cov.scaled #save the covariance matrix

# PERFORMING THE SENSITIVITY ANALYSIS

HonestDiD::createSensitivityResults(betahat = betahat,
                                    sigma = sigma,
                                    numPrePeriods = 5,
                                    numPostPeriods = 2,
                                    Mvec = seq(from = 0, to = 0.05, by =0.01))

Note that performing the createSensitivityResults function will cause an error here, which I think is because of the size of the matrices that are passed into the arguments sigma and betahat. I omitted any information from the controls as such and did not receive an error message:

## FIXING THE COEFFICIENTS AND STANDARD ERROR MATRICES
betahat <- summary(twfe_results)$coefficients[1:7] #save the coefficients
sigma <- summary(twfe_results)$cov.scaled[1:7,1:7] #save the covariance matrix


HonestDiD::createSensitivityResults(betahat = betahat,
                                    sigma = sigma,
                                    numPrePeriods = 5,
                                    numPostPeriods = 2,
                                    Mvec = seq(from = 0, to = 0.05, by =0.01))

I am wondering if this is the process that should be taken to correctly do this analysis. If so, is it possible that the function could check for controls and omit them?

Thanks!