Comments (11)
To be clear, at my test, the bloom filter hash value test totally elapsed 5.5s , the vptest
instruction code takes 4.94s, the vptest
occupies the largest performance.
from libfilter.
How large was your filter? What processor were you using?
from libfilter.
I runed the tpch-300 Q7, the vtune result is captured by running the Q7 circularly.The result shows that the chocke point is the vpptest
. The largest filter size is 1048576(calculated by 1ull << (_log_num_buckets + LOG_BUCKET_BYTE_SIZE)
). My processor is intel skylake. The cpu cache size is 33792 KB.
from libfilter.
Yes, I think pre-fetching might help, if you're doing a lot of lookups in bulk. I can take that as a feature request.
What DBMS are you using?
from libfilter.
alicloud's hologres
from libfilter.
I've tested out prefetching and I can't seem to get a performance benefit on my machine from filters as small as 1MiB, though it's close for larger filters.
from libfilter.
@jbapple https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#text=_mm256_testc_si256&ig_expand=7216 the doc shows that vptest
has a high latency at the skylake architecture cpu.
from libfilter.
what about replace it with two operations: a bitwise-and operation and then a popcount operation
from libfilter.
I implemented the prefetch logic,got a minor performance improvement.
from libfilter.
I implemented the prefetch logic,got a minor performance improvement.
Great! Can you show me how you did it, since our results differed? I'm wondering if I did something slow.
from libfilter.
I am pleasure to share. But the prefetch is implemented outside this lib and is related to my own codes. Basic logic is to hold 32 number of hash values which are prefetched first(a prefetch is to prefetch a bucket), then to do the actual test of the held hash values. If you share what you wrote ,I can point it out.
from libfilter.
Related Issues (20)
- Submit to conan.io
- Ensure binary compatibility between Java and C filters
- Rust bindings
- Sort coupled (aka static) filter before peeling
- Support multi-threading HOT 4
- SIMD for cuckoo filters
- SIMD for peeling filters
- API documentation
- In Java XorFilter, migrate from arrays of structs to structs of arrays
- Support Union and Intersection for more filter types
- serialization and cross language/platform support HOT 7
- Benchmark for different filters HOT 3
- split block bloom filters implementation HOT 1
- Save the filter to SSD
- Java Taffy Block Filter bug
- [Java] Block filter has high fpp HOT 1
- Rename XorFilter to StaticFilter in Java
- Submit to PyPI HOT 1
- Submit to pkg.go.dev
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libfilter.