Tools for the curating of data.
- Python >= 3.8
from datacurators.dataclass_curator import dataclass_curator
Accepts any dataclass and dictionary. Returns a dictionary that will unpack into dataclass without error.
This is done by matching the dataclass attributes with the dictionary's keys. Missing key/ values, by default, are added as None
. Unexpected values are removed from the result.
- Does not enforce type-hints
- Does not override attribute defaults
- Default value for missing keywords can be set per call
- Behavior of adding/removing values can be toggled
- Optional logging of changes made as INFO level logs
import dataclasses
from datacurators.dataclass_curator import dataclass_curator
@dataclasses.dataclass
class Example:
first_value: str
second_value: int
third_value: bool = False
sample_input = {
some_value: "This doesn't match",
second_value: 42,
}
# This will cause a TypeError as we are:
# - Missing `first_value`
# - some_value is an unexpected keyword
mydataclass = Example(**sample_input)
# This will not error as we:
# - Add a None default to `first_value`
# - Remove `some_value`
mydataclass = Example(**dataclass_curator(Example, sample_input))
from datacurators.nested_update import nested_update
Merge two mappings, inluding nested mappings, returning a new result.
This is done by starting with a copy of map1. Key/values that are unique to map2 are added to map1. Key/values that exist in both are updated with map1[key] = map2[key]. Nested mapping structures have the same logic applied, recursively.
Note: Collections of mappings, such as a list of dicts, must be handled with your own implemented logic. This is because there is no way to ensure the order in which objects of the list are updated.
import json
from datacurators.nested_update import nested_update
STEP01 = {"name": "preocts", "type": {}}
STEP02 = {
"name": "Preocts",
"type": {
"style": "egg",
"size": "smol",
},
"likes": [
{"id": 0},
{"id": 1},
],
}
STEP03 = {
"type": {
"style": "Egg",
"shell": "thicc",
},
"likes": [
{"id": 0},
],
}
step_one = nested_update(STEP01, STEP02)
step_two = nested_update(step_one, STEP03)
print(json.dumps(step_one, indent=2))
print(json.dumps(step_two, indent=2))
{
"name": "Preocts",
"type": {
"style": "egg",
"size": "smol"
},
"likes": [
{
"id": 0
},
{
"id": 1
}
]
}
{
"name": "Preocts",
"type": {
"style": "Egg",
"size": "smol",
"shell": "thicc"
},
"likes": [
{
"id": 0
}
]
}
It is highly recommended to use a venv
for installation. Leveraging a venv
will ensure the installed dependency files will not impact other python projects.
Clone this repo and enter root directory of repo:
$ git clone https://github.com/preocts/data_curators
$ cd data_curators
Create and activate venv
:
# Linux/MacOS
python3 -m venv venv
. venv/bin/activate
# Windows
python -m venv venv
venv\Scripts\activate.bat
# or
py -m venv venv
venv\Scripts\activate.bat
Your command prompt should now have a (venv)
prefix on it.
Install editable library and development requirements:
# Linux/MacOS
pip install -r requirements-dev.txt
pip install --editable .
# Windows
python -m pip install -r requirements-dev.txt
python -m pip install --editable .
# or
py -m pip install -r requirements-dev.txt
py -m pip install --editable .
Install pre-commit hooks to local repo:
pre-commit install
pre-commit autoupdate
Run tests
tox
To exit the venv
:
deactivate
This repo has a Makefile with some quality of life scripts if your system supports make
.
install
: Clean all artifacts, update pip, install requirements with no updatesupdate
: Clean all artifacts, update pip, update requirements, install everythingbuild-dist
: Build source distribution and wheel distributionclean-pyc
: Deletes python/mypy artifactsclean-tests
: Deletes tox, coverage, and pytest artifactsclean-build
: Deletes build artifactsclean-all
: Runs all clean scripts