Comments (4)
Thanks mitja for the request. I can see how this could be useful.
It's probably the case that most of the in-memory indexes, could be rebuilt in
memory faster than they could be deserialized from disk. So only the objects in
the collection would need to be serialized/deserialized, and then the indexes
re-added. It sounds like a case for adding a readObject() method.
In the meantime, say with version 1.0.3, you could use the
IndexedCollectionSerializer class below, to serialize an indexed collection.
The main catch, is when deserialized, you *need to re-add the indexes!*. See
the SerializerDemo class below for an example.
I'll think about better serialization support for the next release. Thanks!
--------------------------------------------------------------------------------
package com.googlecode.cqengine;
import com.googlecode.cqengine.index.radix.RadixTreeIndex;
import java.io.File;
public class SerializerDemo {
public static void main(String[] args) {
// *************** Build some collection... ***************
IndexedCollection<Foo> myCollection = CQEngine.newInstance();
addIndexesToMyCollection(myCollection);
// Add some objects...
myCollection.add(new Foo("bar"));
myCollection.add(new Foo("baz"));
// *************** Serialize the collection... ***************
IndexedCollectionSerializer.serialize(myCollection, new File("foo.dat"));
// *************** Deserialize the collection... ***************
IndexedCollection<Foo> myDeserializedCollection = IndexedCollectionSerializer.deserialize(new File("foo.dat"));
// Need to add indexes again to the deserialized collection!!...
addIndexesToMyCollection(myDeserializedCollection);
// ************ myDeserializedCollection should now have the same state as myCollection *******
}
static void addIndexesToMyCollection(IndexedCollection<Foo> indexedCollection) {
indexedCollection.addIndex(RadixTreeIndex.onAttribute(Foo.NAME));
}
}
--------------------------------------------------------------------------------
package com.googlecode.cqengine;
import com.googlecode.cqengine.attribute.Attribute;
import com.googlecode.cqengine.attribute.ReflectiveAttribute;
import java.io.Serializable;
public class Foo implements Serializable {
public final String name;
Foo(String name) {
this.name = name;
}
public static final Attribute<Foo, String> NAME = ReflectiveAttribute.forField(Foo.class, String.class, "name");
}
--------------------------------------------------------------------------------
package com.googlecode.cqengine;
import java.io.*;
import java.util.ArrayList;
import java.util.List;
public class IndexedCollectionSerializer {
public static <O> void serialize(IndexedCollection<O> indexedCollection, File destination) {
OutputStream os = null;
try {
os = new BufferedOutputStream(new FileOutputStream(destination));
List<O> objectsList = new ArrayList<O>(indexedCollection);
ObjectOutputStream oos = new ObjectOutputStream(os);
oos.writeObject(objectsList);
oos.flush();
}
catch (Exception e) {
throw new IllegalStateException(e);
}
finally {
if (os != null) {
try { os.close(); } catch (Exception ignore) {}
}
}
}
public static <O> IndexedCollection<O> deserialize(File source) {
ObjectInputStream ois = null;
try {
ois = new ObjectInputStream(new BufferedInputStream(new FileInputStream(source)));
@SuppressWarnings({"unchecked", "UnnecessaryLocalVariable"})
List<O> objectsList = (List<O>) ois.readObject();
return CQEngine.copyFrom(objectsList);
}
catch (Exception e) {
throw new IllegalStateException(e);
}
finally {
if (ois != null) {
try { ois.close(); } catch (Exception ignore) {}
}
}
}
}
--------------------------------------------------------------------------------
Original comment by [email protected]
on 6 Feb 2013 at 6:55
- Changed state: Accepted
- Added labels: Type-Enhancement
- Removed labels: Type-Defect
from cqengine.
Niall,
first of all thank you for very fast response. Unfortunately this is not my use
case. I would use to store IndexedCollection to some Memcache engine(needs to
implement Serializable) and not to file. So due the fact that you stated that
serializing indexes would take to much time, it would be helpful to avoid
"CQEngine.copyFrom" step, so that IndexedCollection could be "stored" directly
without copying and again retrieved without copying, even if indexes must be
added again.
Otherwise I need to say: Great project!!
Original comment by [email protected]
on 6 Feb 2013 at 7:18
from cqengine.
Usually memcache is used to serialize a single object per key. Whereas in this
case you will store an entire collection against a single key? Will this be
retrieved on application startup, or are you planning to do this for each
request?
Memcache is basically "remote RAM". Which indeed is usually faster than local
disk, but slower than local RAM. It might be worth looking at a distributed
cache which supports local RAM with distributed eviction, instead of going
across the network every time. Also take a look at Kryo as an alternative to
Java serialization. It is much faster and does not require classes to implement
the Serializable interface. I've not tested it with IndexedCollection, but I've
had good results from it in the past.
Nonetheless, even if Kryo works with IndexedCollection right now, there are
still a few optimizations to CQEngine which could improve serialization. I will
add support to serialize the indexed collection without copyFrom in the next
release.
Original comment by [email protected]
on 8 Feb 2013 at 12:54
from cqengine.
Not exactly what was requested, but IndexedCollection can now be persisted to
off-heap memory or to a file on disk. It does not rely on Java serialization.
Original comment by [email protected]
on 20 Apr 2015 at 8:25
- Changed state: Fixed
from cqengine.
Related Issues (20)
- Auto-generate attributes HOT 2
- FilteringIterator.hasNext advances iterator every time it is called. HOT 6
- Recommendation on ID indexes HOT 1
- And.calcHashCode being performance Bottlneck HOT 11
- Set.add semantics is not perfect. HOT 4
- Set remove() and removeAll() doesn't follow substitutability principle. HOT 4
- Add listeners (observers) to IndexedCollection HOT 5
- Add support for query "dialects" like SQL HOT 2
- Documentation for selective READ_COMMITTED transaction isolation using RW lock HOT 2
- Support lock-free READ_COMMITTED transaction isolation via multiple buffering HOT 2
- DeduplicationStrategy.MATERIALIZE broken in 1.2.4 HOT 1
- Modify Or/And to allow only one child. HOT 2
- Comparing two objects? HOT 1
- ObjectLockingIndexedCollection's StripedLock does not handle negative hashcodes HOT 1
- Multiversion concurrency control(MVCC) should be implemented for transaction isolation HOT 2
- Query CONSISTENTLY slower with the addition of NavigableIndex HOT 1
- Provide the way to determine how much memory used by index HOT 2
- Attribute#getValues to return Iterable HOT 1
- deduplication not return duplicate objects from the collection HOT 3
- IllegalArgumentException when ResultSet.getMergeCost() changes during query HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cqengine.