PanelOLS with two-way fixed effects / double clustered std. errors for a panel of millions of observations

[bashtage]

PanelOLS with two-way fixed effects / double clustered std. errors for a panel of millions of observations

[bashtage]

Short term solution is to use one of entity_effects=True and pass the time effect as an other_effects value. This triggers a somewhat slower approach that never builds dummy variable matrices. But it also avoids lots of memory allocation.

Longer term solution is to automatically switch to the slow method when the dimensions of the effects is large, and to expose a user option to override the automatic choice.

[matbuechner]

Thanks!

[bashtage]

This fixed in master now. Can use low_memory=True in fit or it will automatically use low memory if the dummy variable matrix is > 1GiB (which yours easily is ~10M by 4000 = 320GiB)