Invalid polars.lit causes panic, ungraceful termination or hang

[mesner]

Invalid polars.lit causes panic, ungraceful termination or hang

[stinodego]

While the panic exception itself apparently does not subclass Exception intentionally, Polars should not panic here. Broadcasting a literal of type List(Object) is the culprit here:

import polars as pl

df = pl.DataFrame({"x": [1, 2]}).with_columns(pl.lit([object()]))

[mesner]

Ok I think this is related to #5606
Adding a catch for polars panic error results in what I expect.
Rethrowing as an exception causes the multiprocessing to finish gracefully.

except pl.PolarsPanicError as ex:
     print(f"PolarsPanicError error: {ex}")
     raise ValueError("PolarsPanicError error: {ex}")

This seems to be a trap. If end users are never supposed to see panics, then a catch-all and rethrow seems appropriate (perhaps for production builds or gated behind an env var). If users are expected to see panics, then having to catch a custom error seems inconvenient and error prone. Additionally, I've not confirmed, but it seems that generic error-catching assumptions of the multiprocessing library might extend to other libraries as well, and polars' (or any py03 library?) compatibility with them is degraded.

I admit that this could be argued as expected behavior and not a bug, but I still contend that it is not ideal.

I'll leave it open for comments. Many thanks for a great library.

[stinodego]

Simpler reproducer:

import polars as pl

try:
    raise pl.PolarsPanicError()
except Exception:
    print("hello")
# pyo3_runtime.PanicException

Indeed, I think this should be a proper Exception. The exception does subclass BaseException, so the following code will work:

import polars as pl

try:
    raise pl.PolarsPanicError()
except BaseException:
    print("hello")
# hello

The Python docs mention that exceptions should generally prefer to subclass Exception rather than BaseException.

programmers are encouraged to derive new exceptions from the Exception class or one of its subclasses, and not from BaseException.

Perhaps this is a bug in PyO3. I'll check with them to see if there is anything we can do. We cannot really solve this in Polars I think.

[stinodego]

I reported this to PyO3: PyO3/pyo3#3918

I will close this for now as I don't think there's anything Polars can do about this.

[mesner]

@stinodego Any response to my interpretation in PyO3/pyo3#3918 (comment) ?

Edit: Thanks for the response and for looking into this!