chekos/pypums

Download Public Use Micro Sample (PUMS) data files from the US Census Bureau's FTP server.

pypums

Download Public Use Micro Sample (PUMS) data files from the US Census Bureau's FTP server.

Usage

To use PyPUMS in a project:

or as a CLI

Installation

Install this library using pip:

pip install pypums

Usage

Usage instructions go here.

Development

To contribute to this library, first checkout the code. Then create a new virtual environment:

cd pypums
python -m venv venv
source venv/bin/activate

Or if you are using pipenv:

pipenv shell

Now install the dependencies and test dependencies:

pip install -e '.[test]'

To run the tests:

pytest

📃 Citation

@misc{pypums,
  author = {chekos},
  title = {Download Public Use Micro Sample (PUMS) data files from the US Census Bureau's FTP server.},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/chekos/pypums}}
}

pypums's Issues

Link to read docs broken.

The link to read the docs is broken.

Add 2020 data

🚀 Feature Request

The census did not release 2020 1-year ACS data because of COVID but they have it under the experimental section (with experimental weights). There is a 5-year PUMS dataset though.

📎 Additional context

Add the experimental flag.

`--sample-unit` causes an error

🐛 Bug Report

🔬 How To Reproduce

Steps to reproduce the behavior:

1. run pypums download-acs --year 2018 --state alaska --sample-unit household on mybinder link

Code sample

pypums download-acs --year 2018 --state alaska --sample-unit household

Environment

• OS: [e.g. Linux / Windows / macOS]
• Python version, get it with:

python --version

Linux. python 3.7

Screenshots

📈 Expected behavior

Download a household-level dataset

📎 Additional context

Error:

jovyan@jupyter-chekos-2dpypums-2dgdr3re8j:~$ pypums download-acs --year 2018 --state alaska --sample-unit household
Traceback (most recent call last):
  File "/srv/conda/envs/notebook/bin/pypums", line 8, in <module>
    sys.exit(cli())
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pypums/cli.py", line 94, in download_acs
    _download_data(url, "acs", data_directory, extract)
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/pypums/utils.py", line 125, in _download_data
    total = int(response.headers["Content-Length"])
  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/httpx/_models.py", line 993, in __getitem__
    raise KeyError(key)
KeyError: 'Content-Length'

change CLI assumed data directory

Description

CLI assumed data directory to be ../data/ this assumption holds if we're using pypums in a Jupyter notebook, for example, in the notebooks folder but not if we're using it as a CLI.

Change

Should be changed to ./data/ for the CLI

Move package to hyper-modern-python-package template

Description

The template: https://github.com/TezRomacH/python-package-template would be a big improvement.

From their readme:

For your development we've prepared:

• Supports for Python 3.7 and higher.
• Poetry as the dependencies manager. See configuration in pyproject.toml and setup.cfg.
• Power of black, isort and pyupgrade formatters.
• Ready-to-use pre-commit hooks with formatters above.
• Type checks with the configured mypy.
• Testing with pytest.
• Docstring checks with darglint.
• Security checks with safety and bandit.
• Well-made .editorconfig, .dockerignore, and .gitignore. You don't have to worry about those things.

For building and deployment:

• GitHub integration.
• Makefile for building routines. Everything is already set up for security checks, codestyle checks, code formatting, testing, linting, docker builds, etc. More details at Makefile summary).
• Dockerfile for your package.
• Github Actions with predefined build workflow as the default CI/CD.
• Always up-to-date dependencies with @dependabot (You will only need to setup free plan).
• Automatic drafts of new releases with Release Drafter. It creates a list of changes based on labels in merged Pull Requests. You can see labels (aka categories) in release-drafter.yml. Works perfectly with Semantic Versions specification.

For creating your open source community:

• Ready-to-use Pull Requests templates and several Issue templates.
• Files such as: LICENSE, CONTRIBUTING.md, CODE_OF_CONDUCT.md, and SECURITY.md are generated automatically.
• Stale bot that closes abandoned issues after a period of inactivity. (You will only need to setup free plan). Configuration is here.
• Semantic Versions specification with Release Drafter.

Moving to simonw/python-lib

Reference: https://github.com/simonw/python-lib/

🔈 Motivation

Moving to the hyper-modern-python-package template was a mistake. I actually do not use poetry at all and it comes with so much extra automation that I did not set up that I find it hard to develop. This package is small enough that I can rewrite it from scratch and set it up in a way that's actually easier to develop.

fix __repr__ for invalid surveys

• pypums version: 0.0.4
• Python version: 3.7
• Operating System: MacOS

Description

When creating an instance of ACS with an invalid survey type (3-year in 2016, for example).
There is a message displayed that lets the user know that there is no 3-year ACS for 2016 and that it'll default to 5 year. However, the repr still shows 3-year. It should change to 5-year if that's the defaulted value to avoid confusion.

year < 2006 still uses "1-Year" in URL

• pypums version: 0.0.4
• Python version: 3.6+
• Operating System: MacOS

Description

When constructing the ACS()._SURVEY_URL - ACSs from before 2006 should not have "1-Year" in their url. The if statement in pypums.surveys skips if (by default) the current value is 1-year.