Custom check erroneously passes when validating `pl.LazyFrame`

[philiporlando]

Custom check erroneously passes when validating `pl.LazyFrame`

[cosmicBboy]

See the docs here https://pandera.readthedocs.io/en/latest/polars.html#error-reporting

This is intended behavior: LazyFrame validation will only to schema-level checks (so as not to materialize the data in a lazy method chain). Currently, pandera assumes that all custom checks operate on data. You can force data-level checks by explicitly setting export PANDERA_VALIDATION_ENABLED=SCHEMA_AND_DATA.

[cosmicBboy]

Gotcha, yeah looks like a bug, looking.

[cosmicBboy]

Is this a duplicate of #1565?

[philiporlando]

See the docs here https://pandera.readthedocs.io/en/latest/polars.html#error-reporting

This is intended behavior: LazyFrame validation will only to schema-level checks (so as not to materialize the data in a lazy method chain). Currently, pandera assumes that all custom checks operate on data. You can force data-level checks by explicitly setting export PANDERA_VALIDATION_ENABLED=SCHEMA_AND_DATA.

This is super helpful and makes total sense. Thanks for the feedback.

[philiporlando]

Is this a duplicate of #1565?

I don't think so. The error that I'm experiencing in #1565 is specific to pl.DataFrame.

[cosmicBboy]

@philiporlando would it make sense to add some logging at validation time to explicitly say what types of checks are being run? If so, would it make sense as logging.info, debug or something else?

[philiporlando]

@philiporlando would it make sense to add some logging at validation time to explicitly say what types of checks are being run? If so, would it make sense as logging.info, debug or something else?

I'm in favor of this! At the very least, I think it would be helpful to communicate which data-level checks are ignored whenever a LazyFrame is validated instead of a DataFrame. It might even make sense to log a warning here?

[philiporlando]

Gotcha, yeah looks like a bug, looking.

Thank you for looking into it!