Comments (2)
Actually it should, but only 'named' selections. For instance
ds.plot("x", "y", selection="z > 0")
would not cache anything, but
ds.select("z > 0", name="zpos")
ds.plot("x", "y", selection="zpos")
should cache it, so executing the plot a second time should go faster.
Although caching all selections with some memory limit would be an option, I want to leave it out to keep vaex as simple as possible. I hope this will suffice for the moment.
However, I was testing the performance of this, and found it was not being cached. I fixed this, however, don't expect massive speedups for simple selections. Since having a selection means that the data has to be copied (though this may change in the future), it means a performance penalty for selections.
I'll get back to you on this, in combinations with performance improvements for multiple plots, you may see some speedup, which should also fix #10.
from vaex.
This is working well now (using arctan2 since it's quite expensive to calculate):
ds = vaex.datasets.helmi_de_zeeuw.fetch()
ds.select("arctan2(y,x) > 0")
%%timeit -r5
ds.count(selection=True)
Gives:
10 loops, best of 5: 18.5 ms per loop
While using an unnamed selection (which will not get cached)
%%timeit -r5
ds.count(selection="arctan2(y,x) > 0")
Gives
1 loop, best of 5: 120 ms per loop
6.5x speedup.
from vaex.
Related Issues (20)
- [BUG-REPORT] Printing vaex df after sort running out of memory
- [BUG-REPORT] HOT 1
- [BUG-REPORT] Large Groupby Agg runs out of memory
- Vaex not exporting to file properly inside of a mulitprocessing pool.
- [FEATURE-REQUEST] looking for vaex equivalent of pandas_df.corr(min_periods=100)
- [FEATURE-REQUEST] Getting dtype of columns as they are when rendered in a pandas dataframe?
- [BUG-REPORT] SyntaxError when representing result of a basic operation involving Expression and numpy array, with the array on the right of the operator HOT 1
- How do i overcome this issue? HOT 1
- [BUG-REPORT] why the same dataframe and same groupby(vaex.agg.list) operation, one got error and one got correct if i export to hdf5 and loaded it back
- [BUG-REPORT] jupyter notebook error in tutorial docs
- [BUG-REPORT] HDF5 file remains locked after close
- [BUG-REPORT] MinMaxScaler not working after filtering
- [FEATURE-REQUEST] is it still supported
- [FEATURE-REQUEST] read hdf5 file in C++
- [BUG-REPORT] Docs are down HOT 2
- statistics on 2d grids: control the `bin_centers`
- interactive statistic heatmap / quadrilateral mesh
- Unexpected Discrepancy Between Printed Values and Returned Results in vaex.apply() Function
- Exporting out of memory dataframe to parquet error
- Support Cross-Validation ??
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vaex.