zhenhua-wang/missingdata_dl

MissingData_DL

Author: Zhenhua Wang, Olanrewaju Akande, Jason Poulos and Fan Li

Paper:

Are deep learning models superior for missing data imputation in large surveys? Evidence from an empirical comparison:

Datasets:

• household: https://www2.census.gov/programs-surveys/acs/data/pums/2019/1-Year/
  • processed data(https://drive.google.com/file/d/1j9KHLJeI4JPyjWcQ5r78hxG5493Msg8l/view?usp=sharing)
• spam: https://archive.ics.uci.edu/ml/datasets/Spambase
• letter: https://archive.ics.uci.edu/ml/datasets/Letter+Recognition
• breast: https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)
• credit: https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients
• news: https://archive.ics.uci.edu/ml/datasets/online+news+popularity

Usage

1. Use sampler.py to create samples with MCAR.
2. Use main.py to impute the missing dataset.
3. To evaluate the performance of missing imputation, we first need to calculate the estimands in the poputaion dataset, the complete sample dataset and the imputed data using evaluation/calculate_estimands.py. Next, we use evaluation/evaluate_estimands.py to calcaute the performance metrics.
4. To display the performance metrics, we use plot_figures.py and plot_tables.py.

missingdata_dl's Issues

Would it be possible to include data preparation code?

Hi

I read the corresponding paper on arxiv and I was going to try running the code, but it wasn't obvious how to produce data/house.csv. I downloaded https://www2.census.gov/programs-surveys/acs/data/pums/2019/1-Year/csv_hus.zip, but that obviously needs a fair bit of treatment to be turned into something consistent with your code? I wonder if it might be possible to include the code that does this preprocessing?

As a minor aside, I found the utils/utils.py script had a few indentation issues that stopped the code running for me. Mostly just the docstrings, I think.

Training Query

Hi,
I've been implementing your code, and its been really simple to work with due to the great structure- thank you!
I have a question about the GAIN_v2.py script, I noticed that you're training the first and third layers of the generator, but all three layers of the discriminator. What was the rationale behind not training this central layer? I couldn't find a reference to it in the paper, but may have missed it.
Thanks! Robyn