martinbenes1996/eurostat_deaths

Web Scraper for Eurostat data.

Eurostat

Package is a simple interface for parsing data from Eurostat:

• deaths counts
• population sizes

Download the package with

pip3 install eurostat_deaths

To import and fetch data, simply write

import eurostat_deaths as eurostat

Function deaths() fetches the deaths, function populations() fetches the populations. Their description is in following sections below.

Package is regularly updated. Upgrade your local version typing

pip3 install eurostat_deaths --upgrade

Deaths

from datetime import datetime
import eurostat_deaths as eurostat

data = eurostat.deaths(start = datetime(2019,1,1))

Parameter start sets the start of the data. The end is always now().

You receive per-week data of deaths. Since the total size of the data frame is about 218 MB, call takes more than 15 minutes. The usage of memory is significant.

In the future, module will be reimplemented to use Big Data framework, such as PySpark.

The pandas dataframe is returned.

from datetime import datetime
import eurostat_deaths as eurostat

# does not return, create a file with result
eurostat.deaths(output = "deaths.csv", start = datetime(2019,1,1))

Parameter output = "deaths.csv" causes that the output is saved into file deaths.csv before returning.

One additional setting is chunksize to set the size of chunk, that is processed at a time. The unit used is thousands of rows.

Population

Populations in years for NUTS-2 and NUTS-3 regions can be fetched such as

import eurostat_deaths as eurostat

data = eurostat.populations()

Similarly as in deaths() call, populations() can be parametrized with chunksize (in thousands of lines) and output, forwarding the output to file rather than returning and hence saving time allocating a big data frame in main memory.

import eurostat_deaths as eurostat

# does not return, create a file with result
eurostat.populations(output = True)

Here the data volume is incomparably lower and hence the regular usage to return the data frame is possible.

Credits

Author: Martin Benes.

eurostat_deaths's Issues

Eurostat caching

Data fetching takes too long. The preprocessing and caching should be solved.

Country-wise filtering

Filtering of Eurostat deaths based on country should make the result much more easily reachable and the function call much faster.