moj-analytical-services/dataengineeringutils

A python package containing functions that help manage our data management processes on AWS

data Engineering Utils

A python package containing functions that help manage our data management processes on AWS

To install this package pip install git+git://github.com/moj-analytical-services/dataengineeringutils.git#egg=dataengineeringutils

If you want to update the package then you need to delete it first before reinstalling i.e. run: pip uninstall dataengineeringutils

Warning: This package has the following dependencies:

• numpy
• pandas
• io
• boto3

This package doesn't list its package denpencies because I found errors with io when installing via pip so I have left it blank for now ¯\_(ツ)_/¯

dataengineeringutils's Issues

Where to store partition keys in generic metadata?

At the moment, I've included them in the generic metadata files but this is clearly wrong because the metadata files are supposed to be agnostic to the database being used.

See here:

https://github.com/moj-analytical-services/data_warehouse_database_template/blob/master/metadata/people_data.json

Job run shouldn't fail if glue_resources doesn't exist

Cannot use float in pandas for columns of type int

http://pandas.pydata.org/pandas-docs/stable/gotchas.html#support-for-integer-na
Due to the above gotya, we started by using floats in pandas where the metadata said the column was an int

This solution can't work because it means if we impose metadata types on a pandas dataframe, ints get converted to floats and csvs get outputted with values like 9.0 rather than 9. Hive then throws an error when it sees a value like this which is meant to be an int.

NA support is coming to pandas ints soon (0.24.0) documented here

The principle of least surprise suggests we treat ints as ints in pandas, and then explicitly have to deal with the current problems around NAs.

Now prints None twice

It is sometimes helpful to be told that nothing is happening but when it occurs twice in quick succession soon after starting every glue job it becomes worrysome...

Issue is here:

dataengineeringutils/dataengineeringutils/glue.py

Line 272 in 51d77f7

print(response)

Impose metadata conformance doesn't guarantee strings

robinlinacre [11:11 AM]
little warning/note about pd.to_parquet. If you load a string column into pandas e.g. from a csv, the dtype will be object.
But if that col contains e.g. 2,2,2C,2, then you might end up with a column of mixed types
the dtype will still be object but some of the contents will be floats/ints and some will be string
pd.parquet will fail because columns in parquet have to be a single type
so you have to explicitly convert using df[col] = df[col].to_string()

get_csv_header doesn't strip quotes

Should probably strip all special characters or have a param to do so

meta data error on partitioned data

partition keys are not converted when creating glue meta data catalogue