statisticsnorway/dapla-toolbelt-pseudo

Pseudonymization extensions for Dapla Toolbelt

dapla-toolbelt-pseudo's Issues

500 server error on "sid_fields" - Allow only filling "sid_fields", not demanding "fields"

I have dataset with a single column "fnr", this is a sid-field.
dpp.pseudonymize(data, sid_fields=["fnr"]).json()
Returns
TypeError: pseudonymize() missing 1 required positional argument: 'fields'

Whenever I specify "sid_fields", I get a 500-server error.
dpp.pseudonymize(data, fields=["fnr"], sid_fields=["fnr"])

HTTPError: 500 Server Error: Internal Server Error for url: http://dapla-pseudo-service.dapla.svc.cluster.local/pseudonymize/file

If I only specify "fnr" as a "field", I get the wrong encryption. The goal is to get FPE on the "fnr" column.

Allow depseudonymize to take dataframe like pseudonymize

Currently depseuonymize returns
TypeError: expected str, bytes or os.PathLike object, not DataFrame

Support compression

The dapla-pseudo-service API supports compression, ref:
https://dapla-pseudo-service.staging-bip-app.ssb.no/api-docs/redoc

Users should be able to specify compression accordingly for all three endpoints:

• pseudonymize/file
• depseudonymize/file
• repseudonymize/file

Can't pseudonymize sid_fields only

Remove special handling of metadata from map-sid is

The pseudo-service previously returned two metadata entries when using map-sid. This behaviour has been changed on the server side, so dapla-toolbelt-pseudo no longer needs to handle it.

See https://github.com/statisticsnorway/dapla-toolbelt-pseudo/blob/main/src/dapla_pseudo/v1/result.py#L205

Improve API interaction error handling

If the REST API returns an non-200 status, we could do a better with regards to how these errors are presented to the user.

We should also be explicit about errors that is due to wrong user input (http status 4xx), and unexpected errors that occur on the server side (http status 5xx).

Support partitioned parquet

Parquet files may be partitioned for efficiency. When this happens, the dataset is saved as multiple .parquet files in a directory. To open the dataset, one refers to the path to the directory. This isn't currently supported by dapla-toolbelt-pseudo as we expect a file ending.

Use polars for file IO where possible

Polars Dataframe is now the standard internal data representation for dapla-toolbelt-pseudo because it is more performant than Pandas.

Polars supports IO for:

• CSV
• Parquet

We should use Polars where we can.

Support sending in "empty" to .on_fields()

This is not currently possible, and is forcing me into what I would consider an anti-pattern.

Im using a single process_source-script for a load of files, sometimes they:

• Have both fields for .with_stable_id() AND for .with_papis_compatible_encryption()
• ONLY fields for .with_stable_id()
• ONLY fields for .with_papis_compatible_encryption()
• NO FIELDS for pseudo (but should still pass through Kildomaten)

The anti-pattern consists of having three different codeblocks that use Pseudodata-chains dependant on what types of columns the dataset contains. See the code below. Should be possible to send an empty list or similar to .on_fields(), and then for dapla-toolbelt-pseudo to not send anything for pseudo, just pass the object along. If that was possible, I could reduce this down to a single Pseudo-block, that would be very nice 😎

    if fnr_cols and non_map_cols:
        logging.info("Want to pseudo and map to snr: %s - output path: %s",
                     str(fnr_cols), str(output_path))
        logging.info("Want to pseudo, but not map to snr: %s - output path: %s",
                 str(non_map_cols), str(output_path))
        df = (PseudoData.from_pandas(df)
        .on_fields(*renames_postfix.keys())
        .with_stable_id()
        .on_fields(*non_map_cols)
        .with_papis_compatible_encryption()
        .pseudonymize()
        .to_pandas()
        )
    elif fnr_cols:
        logging.info("Want to pseudo and map to snr: %s\nOutput path: %s",
                     str(fnr_cols), str(output_path))
        df = (PseudoData.from_pandas(df)
        .on_fields(*renames_postfix.keys())
        .with_stable_id()
        .pseudonymize()
        .to_pandas()
        )
    elif non_map_cols:
        logging.info("Want to pseudo, but not map to snr: %s\nOutput path: %s",
                 str(non_map_cols), str(output_path))
        df = (PseudoData.from_pandas(df)
        .on_fields(*non_map_cols)
        .with_papis_compatible_encryption()
        .pseudonymize()
        .to_pandas()
        )
    else:  # No columns to pseudo
        return df