Comments (5)
Found that we accidentally introduced quadratic behavior for the new string type. This is resolved by: #14705
from polars.
Can we get a bisect to see which commit introduced the slow down?
I expect this is the new string type. Isn't reader the file just slower?
from polars.
I do see a difference in performance on linux, but not as drastically as on windows. I believe windows is due to the allocator we compile on windows which is very bad. We have to try to switch to default allocator.
from polars.
I think the difference is due to utf8 validation. This is much more expensive because we cannot do it on the whole buffer anymore:
I will see if we can do something smart here.
from polars.
version 0.20.11 indeed fixes this issue:
# issue exists on random generated data
# 0.20.5 - 253 ms ± 3.68 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 0.20.6 - 2.28 s ± 7.81 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 0.20.10 - 3.03 s ± 19.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# 0.20.11 - 747 µs ± 5.34 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit pl.scan_parquet(data_path / "sample_data_gen_4m.parquet").select(pl.len()).collect()
# no issue in this case
# 0.20.5 - 79.9 ms ± 116 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# 0.20.6 - 114 ms ± 604 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# 0.20.10 - 113 ms ± 545 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# 0.20.11 - 687 µs ± 5.12 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
%timeit pl.scan_parquet(data_path / "sample_data_4m.parquet").select(pl.len()).collect()
Thanks for the quick fix!
from polars.
Related Issues (20)
- Printing a LazyFrame triggers resolving to the IR
- Documentation suggests you can use `<`, `>`, `==`, `<=`, `>=` operators to construct `Expr` in Rust but that's fundamentally not possible.
- cannot find function `as_struct` in this scope HOT 2
- QOL improvements for .rolling
- Include example with function accepting multiple arguments in `Expr.map_batches`
- Documentation issue in `normalize`/`name` parameter from `.value_counts()` method HOT 2
- writing to os.devnull
- `Series[list].explode()` should not return `None` for empty lists HOT 2
- write_database to snowflake with adbc engine spouts context canceled error log
- Cannot tell if hvplot version 0.10.0 >= 0.9.1 HOT 2
- dtype 'Time' gets converted to i64 when collect(streaming=True) is used. HOT 1
- `.agg_groups()` PanicException when not used in a group_by context
- Additional Parameter for json_normalize HOT 5
- fold shouldn't require that acc and exprs share the same dtype
- Adding `descending` parameter to `Expr.over` HOT 5
- polars.LazyFrame.head recommends using fetch()
- Reading large json file error: ComputeError: InputTooLarge at character 0
- Serialize for AnyType has a todo!() HOT 1
- File cache invalidation not triggered for HTTP if size is the same
- Loading parquet written from an Arrow table produces non-deterministic incorrect numbers since 1.2.0 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.