kolanich-libs / transformerz.py Goto Github PK

View Code? Open in Web Editor NEW

A library implementing composable and stackable reversible processor objects for data. !!! Migrated to Codeberg 🏔️ !!!

License: The Unlicense

Python 92.04% Shell 0.50% Jupyter Notebook 7.46%

transformerz.py's Introduction

transformerz.py

~~wheel (from GHA via nightly.link)~~

We have moved to https://codeberg.org/KOLANICH-libs/transformerz.py, grab new versions there.

Under the disguise of "better security" Micro$oft-owned GitHub has discriminated users of 1FA passwords while having commercial interest in success and wide adoption of FIDO 1FA specifications and Windows Hello implementation which it promotes as a replacement for passwords. It will result in dire consequencies and is competely inacceptable, read why.

If you don't want to participate in harming yourself, it is recommended to follow the lead and migrate somewhere away of GitHub and Micro$oft. Here is the list of alternatives and rationales to do it. If they delete the discussion, there are certain well-known places where you can get a copy of it. Read why you should also leave GitHub.

Just a set of composable processor objects that can be stacked, and path can be automatically routed.

Each class/object has 2 members of type type:

tgtType
srcType

and 2 functions

process - converts a value of srcType to the tgtType. Should parse the data from the representation useful for storing on disk.
unprocess - converts a value of tgtType to srcType Should serialize the data from the representation useful for storing on disk. .

The names of functions are inherited from kaitai.process library (Kaitai Struct is a parsing framework, so process historically means parsing), so are some classes (and I hope to get the stuff from this package merged supported by KS somewhen).

There are 3 base classes:

TransformerBase - for objects with srcType and tgtType hardcoded in class definitions or available as props
Transformer - for objects with srcType and tgtType stored in slots
FileTransformer - for transformations when unprocessed form can be stored in a file with a well-known extension and possible MIME type.
BinaryProcessor - adapter for Kaitai Struct stuff

There are transformers are of different types and reside in different submodules:

.serialization - packages to serialize various objects
- .json.jsonSerializer - Uses ujson if it is available which is faster than built-in json module.
- .bson.bsonSerializer - Available if pymongo is installed.
- .msgpack.msgpackSerializer - Available if a package for MsgPack serialization is installed.
- .cbor.cborSerializer - Available packages for CBOR serialization: either cbor or cbor2 - are installed.
- .pon.ponSerializer - "Python Object Nonation" - stuff like JSON that can be safely evaluated using literal_eval
.processors - process binary data. This module contains the adapters allowing to use the stuff written to be used in process attr in Kaitai Struct specs.
.compression - packages to compress binary data. Take various params.
.text - convert text to bytes and back
.struct - parses data to tuples and back using struct.Struct. Numbers binary representations also go here. But not all. Some cannot be parsed by struct, so they go to ...
.numpy - parsing and serializing arrays of numbers using numpy machinery. Mostly needed for IEEE 751 floats not built into python.