Comments (5)
While the panic exception itself apparently does not subclass Exception
intentionally, Polars should not panic here. Broadcasting a literal of type List(Object)
is the culprit here:
import polars as pl
df = pl.DataFrame({"x": [1, 2]}).with_columns(pl.lit([object()]))
from polars.
Ok I think this is related to #5606
Adding a catch for polars panic error results in what I expect.
Rethrowing as an exception causes the multiprocessing to finish gracefully.
except pl.PolarsPanicError as ex:
print(f"PolarsPanicError error: {ex}")
raise ValueError("PolarsPanicError error: {ex}")
This seems to be a trap. If end users are never supposed to see panics, then a catch-all and rethrow seems appropriate (perhaps for production builds or gated behind an env var). If users are expected to see panics, then having to catch a custom error seems inconvenient and error prone. Additionally, I've not confirmed, but it seems that generic error-catching assumptions of the multiprocessing library might extend to other libraries as well, and polars' (or any py03 library?) compatibility with them is degraded.
I admit that this could be argued as expected behavior and not a bug, but I still contend that it is not ideal.
I'll leave it open for comments. Many thanks for a great library.
from polars.
Simpler reproducer:
import polars as pl
try:
raise pl.PolarsPanicError()
except Exception:
print("hello")
# pyo3_runtime.PanicException
Indeed, I think this should be a proper Exception. The exception does subclass BaseException
, so the following code will work:
import polars as pl
try:
raise pl.PolarsPanicError()
except BaseException:
print("hello")
# hello
The Python docs mention that exceptions should generally prefer to subclass Exception
rather than BaseException
.
programmers are encouraged to derive new exceptions from the Exception class or one of its subclasses, and not from BaseException.
Perhaps this is a bug in PyO3. I'll check with them to see if there is anything we can do. We cannot really solve this in Polars I think.
from polars.
I reported this to PyO3: PyO3/pyo3#3918
I will close this for now as I don't think there's anything Polars can do about this.
from polars.
@stinodego Any response to my interpretation in PyO3/pyo3#3918 (comment) ?
Edit: Thanks for the response and for looking into this!
from polars.
Related Issues (20)
- Casting a column to pl.Categorical is way slower than pandas (10-20x) HOT 2
- Minimal memory usage tests for read_ipc read_ipc_stream
- Forward_fill() and backward_fill() is about 25% slower in polars compared to pandas' counterparts HOT 6
- Rust code examples missing on page /user-guide/io/cloud-storage/#scanning-from-cloud-storage-with-query-optimisation
- Offset_by is about 4 times slower in polars compared to pandas' counterpart HOT 2
- `filter` + `arg_max` + `over` producing non-deterministic junk values HOT 1
- pyo3_runtime.PanicException: python function failed: PyErr { type: <class 'TypeError'>, value: TypeError("'list' object is not callable"), traceback: None } HOT 1
- support expressions in `Frame.unique()` HOT 2
- Read Options for Calamine HOT 1
- Support for pl.List('*') HOT 4
- See the polars df in Pycharm HOT 5
- schema_overrides failing HOT 4
- .over() performs quite slow in given sample HOT 4
- `.backward_fill()` does not consider `np.nan` to be invalid HOT 2
- from_arrow.consuming large memory HOT 2
- Inconsistent behavior with dataframe level arithmetic when using Python's `sum` HOT 1
- `read_csv` ignores `skip_rows_after_header` when `use_pyarrow=True` HOT 1
- group_by sum agg returns nulls for decimal columns in dataframes with 1000+ rows HOT 2
- Join on struct fails with "not implemented", but on struct + other column fails silently HOT 1
- Add `mode` argument in `pl.DataFrame.write_csv` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.