Comments (4)
Hmm... The usize
should be changed to u64
.
from polars.
Hi @ritchie46, not sure if I understand, usize
is architecture agnostic right? Which 'problematic' use of usize
are you exactly referring to in the code?
from polars.
I see that the offsets
vector in RowsEncoded
is casted later to i64
(after checking the types) and this causes the problem. Is your suggestion to change the type of offsets
to Vec<u64>
, such that they take the same number of bytes on memory?
from polars.
I quickly checked the impact of the above, and the code becomes a bit less appealing using u64
as it will also require changes on encode.rs
casting usize
to u64
. What do you think of the following, where offset<usize>
(naturally) remains as-is:
let offsets = if std::mem::size_of::<usize>() % std::mem::size_of::<i64>() == 0 {
// We can safely convert usize to i64
bytemuck::cast_vec::<usize, i64>(offsets)
} else {
offsets.iter().map(|&i| i as i64).collect()
};
It checks if it can simply convert, or in worst-case (for wasm) it will have some performance impact because it needs to cast the encoded rows.
from polars.
Related Issues (20)
- Wrong result when filtering categorical using `.is_in` in `scan_parquet` HOT 2
- `.over()` fails with `.get`. HOT 6
- Don't disable coalesce when joining on expressions HOT 2
- entered unreachable code when using collect_schema and concat HOT 1
- Use of write_ipc with file=None
- Examples of using `scan_csv` with cloud URIs HOT 2
- Consistency in check_exact between assert_frame_equal and polars.DataFrame.equals
- `compat_level` documentation is unclear
- Request to return LazyFrame for pl.from_arrow and more HOT 2
- rolling_corr giving inconsistent results HOT 1
- `test_read_database_cx_credentials` expected exception does not bifurcate over correct Python version HOT 1
- In `scan_parquet()`, `include_file_paths` returns twice the same column
- In `bin.size()`, the `unit` parameter is not described HOT 1
- `.implode().list` method is not valid before `.over`. HOT 3
- Upgrading to 0.42 breaks compilation HOT 4
- Expr `.implode().get().over()` has strange results. HOT 3
- upsample not working if arguments are only sorted within group HOT 1
- Behaviour change of Expr.list.drop_nulls for structs with all None fields HOT 1
- `.first` behind `.sort_by.slice` gets wrong result.
- `Expr.shuffle` uses different order per column HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.