Comments (1)
Thanks for the report but I'm afraid it's not a bug...
- You are not actually measuring query performance in this test. You are timing
how long it takes CQEngine to return a ResultSet, but queries are evaluated
*lazily* so you need to actually iterate through the results to make CQEngine
do any work.
Here you see a slight increase in the time it takes CQEngine to return a
ResultSet when you have an index in place, because CQEngine is doing some quick
calculations to work out if it will use the index or not. If there are no
indexes available then it doesn't need to make this calculation. But this would
be a negligible computation in normal query processing.
- I don't know if you actually ran this test with COUNT = 20 as in the source
code, but 20 objects is way too few for a benchmark. I assume you tested with
more objects and I tested with 10,000 and 100,000 objects anyway.
- All of your 'between' queries match 50% of the collection. Indexes can make
finding a needle in a haystack a fast operation. But here, you are selecting
half of the haystack :)
By default indexes like NavigableIndex are optimized for finding needles in the
haystack - the buckets are very small so that fine-grained queries can be
evaluated without filtering.
If we have a collection of 10,000 objects, and your typical queries will match
5,000 objects, then actually it means the query will span 5,000 buckets, and
there is some overhead in jumping between the buckets.
So you should configure the index with a Quantizer to reduce the number of
buckets. This will improve performance for coarse-grained queries (and also
reduce memory overhead), but it will reduce performance for fine-grained
queries. So it's about tuning the index based on the granularity of your
expected or typical queries.
I ran the test, with:
----------------------
COUNT = 100000;
...
items.addIndex(NavigableIndex.withQuantizerOnAttribute(
IntegerQuantizer.withCompressionFactor(1000),
Item._x
));
...
Iterables.count(items.retrieve(q[j])); // <- iterates the ResultSet
----------------------
Results were (taking output lines 38 and 41 as examples):
38: Q0: 3167000 // 'equal' query, without NavigableIndex
38: Q0: 246000 // 'equal' query, with NavigableIndex, 13 times faster
41: Q3: 7127000 // 'between' query, without NavigableIndex
41: Q3: 1731000 // 'between' query, with NavigableIndex, 4 times faster
I'll close this issue as Invalid, please don't take offence at that but I don't
think this is a bug. If you find other issues let me know as the more eyes on
the code the better. The discussion forum is also available:
https://groups.google.com/forum/#!forum/cqengine-discuss
Original comment by [email protected]
on 30 May 2014 at 11:51
- Changed state: Invalid
from cqengine.
Related Issues (20)
- Auto-generate attributes HOT 2
- FilteringIterator.hasNext advances iterator every time it is called. HOT 6
- Recommendation on ID indexes HOT 1
- And.calcHashCode being performance Bottlneck HOT 11
- Set.add semantics is not perfect. HOT 4
- Set remove() and removeAll() doesn't follow substitutability principle. HOT 4
- Add listeners (observers) to IndexedCollection HOT 5
- Add support for query "dialects" like SQL HOT 2
- Documentation for selective READ_COMMITTED transaction isolation using RW lock HOT 2
- Support lock-free READ_COMMITTED transaction isolation via multiple buffering HOT 2
- DeduplicationStrategy.MATERIALIZE broken in 1.2.4 HOT 1
- Modify Or/And to allow only one child. HOT 2
- Comparing two objects? HOT 1
- ObjectLockingIndexedCollection's StripedLock does not handle negative hashcodes HOT 1
- Multiversion concurrency control(MVCC) should be implemented for transaction isolation HOT 2
- Provide the way to determine how much memory used by index HOT 2
- Attribute#getValues to return Iterable HOT 1
- deduplication not return duplicate objects from the collection HOT 3
- IllegalArgumentException when ResultSet.getMergeCost() changes during query HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cqengine.