tzaeschke / zoodb Goto Github PK

View Code? Open in Web Editor NEW

56.0 9.0 9.0 13.82 MB

ZooDB Object Database

License: Apache License 2.0

Shell 0.01% Java 99.99%

jdo database java db4o object-database jpa jta

zoodb's People

Contributors

Stargazers

Watchers

Forkers

oserb jonasnick matiasnamendola nickcharsley pk-codebox-evo slavap ruixj mojo2012 nono9527

zoodb's Issues

Provide database format independent XML export/import facility

An XML export/import facility is required in order to allow migration of data between databases with different file format (old and new versions of the DBMS).
Human readable XML is not a priority, thus the XML can for now contain hexadecimal values to represent data.

Make caching behaviour configurable

Let users choose between strong(pinning), soft and weak caching

Ensure uniqueness of generic objects and handles

Currently, getting a handle creates a new handle and new generic object with each call, independent of whether there are already handles/GOs for the given object ID.

getExtent() fails in schema auto-create mode if class is not yet auto-created

getExtent() fails in schema auto-create mode if class is not yet auto-created. It should return an empty collection instead. newQuery() fails accordingly.

javax.jdo.JDOUserException: Class schema not defined: ch.ethz.globis.twitstore.core.IdGenerator
at org.zoodb.jdo.ExtentImpl.(ExtentImpl.java:62)
at org.zoodb.jdo.PersistenceManagerImpl.getExtent(PersistenceManagerImpl.java:169)
at org.zoodb.jdo.PersistenceManagerImpl.getExtent(PersistenceManagerImpl.java:469)
at ch.ethz.globis.twitstore.core.IdGenerator.instance(IdGenerator.java:33)
at ch.ethz.globis.twitstore.core.TwitLogic.handleInsert(TwitLogic.java:261)
at ch.ethz.globis.twitstore.core.TwitLogic.handleParseResult(TwitLogic.java:245)
at ch.ethz.globis.twitstore.core.TwitLogic.storeTwit(TwitLogic.java:140)
at ch.ethz.globis.twitstore.core.TwitstoreImpl.storeTwit(TwitstoreImpl.java:87)
at ch.ethz.globis.twitstore.TwitIdTest.insertTwits(TwitIdTest.java:78)
at ch.ethz.globis.twitstore.TwitIdTest.testIds(TwitIdTest.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

javax.jdo.JDOUserException: Class schema not defined: ch.ethz.globis.twitstore.core.IdGenerator
at org.zoodb.jdo.ExtentImpl.(ExtentImpl.java:62)
at org.zoodb.jdo.PersistenceManagerImpl.getExtent(PersistenceManagerImpl.java:169)
at org.zoodb.jdo.PersistenceManagerImpl.getExtent(PersistenceManagerImpl.java:469)
at ch.ethz.globis.twitstore.core.IdGenerator.instance(IdGenerator.java:33)
at ch.ethz.globis.twitstore.core.StoreManagerImpl.closeStore(StoreManagerImpl.java:131)
at ch.ethz.globis.twitstore.TwitIdTest.cleanup(TwitIdTest.java:41)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

Implement multi-file capability

Reduce file-pos to 32bit? of 48bit?
Split indices to allow concurrent processing? Or prefer single large index, continaing all files? Or simply split the large file into chunks? -> The latter would at least allow smaller file-positons

Query result collections should not allow calling Iterator.remove()

See Test_124_QueryCache.

Separate client-server

Provide tool for comparing databases

Provide a tool that allows comparing two databases for differences in schema and data. This is useful for users as well as developers who can use it in test harnesses.

Mutli-threaded access causes lock exception

Using different threads to open and commit databases causes the following Exception:

java.lang.IllegalMonitorStateException: null
    at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374) ~[na:1.7.0_65]
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260) ~[na:1.7.0_65]
    at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) ~[na:1.7.0_65]
    at org.zoodb.internal.server.DiskAccessOneFile.beginCommit(DiskAccessOneFile.java:505) ~[zoodb.jar:na]
    at org.zoodb.internal.model1p.Node1P.beginCommit(Node1P.java:286) ~[zoodb.jar:na]
    at org.zoodb.internal.Session.processOptimisticVerification(Session.java:192) ~[zoodb.jar:na]
    at org.zoodb.internal.Session.commit(Session.java:128) ~[zoodb.jar:na]
    at org.zoodb.jdo.impl.TransactionImpl.commit(TransactionImpl.java:82) ~[zoodb.jar:na]

Storing an object with a reference to a PersistenceManager should throw an according exception

Instead it throws an NPE (before fix to issue #30 ) or StackOverflowException (after fixing #30).

A test is available in Test_084_SerailizationBugRefToPM.testSerialization().

Query#compile fails with NPE for query on class whose schema does not yet exist

With schema auto-creation on, doing the following before an instance of class Asset is in the database, will lead to a NPE:

this.query = this.pm.newQuery(Asset.class, "id == anId");
this.query.compile();

The issue seems to originate from the following code in QueryImpl#setClass

ZooClassProxy sch = pm.getSession().getSchemaManager().locateSchema(cls, node);
if (sch != null) {
    candClsDef = sch.getSchemaDef();
} else ...

Here, sch is null, which results in candClsDef not being set. That field is initialised to null on creation of QueryImpl. It is then de-referenced when the query is being compiled without a check, leading to the NPE.

Maybe slightly related to issue 36.

Implement generated (de-)serializers

Dirty objects in client cache can cause ConcurrentModificationException

Example: See test in Test_124_QueryCache.java.

query.setFilter() does not reset parameter requirements

This fails:

Query q = pm.newQuery(TestClass.class, "_int == :x");
q.setFilter("_int == 123");
q.execute();

with:

javax.jdo.JDOUserException: Too few arguments given, parameter count: 1
    at org.zoodb.jdo.QueryImpl.checkParamCount(QueryImpl.java:489)
    at org.zoodb.jdo.QueryImpl.execute(QueryImpl.java:567)

But it should not fail.
See
Test_122_QueryBugs.testSetFilterForParameters()

Implement schema evolution

Index updates do not work when update occurs via ZooSchema reflection

See Test_037_SchemaWriting.testGenericObjectStringIndexUpdates():

java.util.NoSuchElementException: Key not found: -9223372036854775808/0
at org.zoodb.jdo.internal.server.index.LLIndexPage.remove(LLIndexPage.java:676)
at org.zoodb.jdo.internal.server.index.LLIndexPage.remove(LLIndexPage.java:669)
at org.zoodb.jdo.internal.server.index.PagedUniqueLongLong.removeLong(PagedUniqueLongLong.java:104)
at org.zoodb.jdo.internal.server.index.PagedUniqueLongLong.removeLong(PagedUniqueLongLong.java:122)
at org.zoodb.jdo.internal.model1p.DataSink1P.updateFieldIndicesGO(DataSink1P.java:285)
at org.zoodb.jdo.internal.model1p.DataSink1P.flushBuffer(DataSink1P.java:164)
at org.zoodb.jdo.internal.model1p.DataSink1P.flush(DataSink1P.java:152)
at org.zoodb.jdo.internal.Session.commitInternal(Session.java:202)
at org.zoodb.jdo.internal.Session.commit(Session.java:92)
at org.zoodb.jdo.TransactionImpl.commit(TransactionImpl.java:87)
at org.zoodb.test.Test_037_SchemaWriting.testGenericObjectStringIndexUpdates(Test_037_SchemaWriting.java:412)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

refresh() and refreshAll() should NOT undelete deleted objects

JDO specifies that PERSISTENT_NEW_DELETED and PERSISTENT_DELETED objects should remain unchanged when they are refreshed and that NO error should be thrown.

Currently ZooDB throws an JdoUserException, see Test_041_TransactionsCommit

Create separate non-JDO API

Clean up GenericObject & GOProxy usage

From TODO:

Generic objects (GOs) and GOProxy (GOPs):
Mixing PC and GOs is impossible, because PCs/SCOs may contain typed references which can not be
set to GO, only to a GOP. Pure GO structures sound feasible, but need to be done carefully,
they need to include DBCollection classes (why? because it is simpler?) and SCOs, which is much
harder to achieve (non-persistent capable generic objects). It also hard because SCOs may actually
be java classes which are already loaded, or custom sub-classes of java classes.
Use of GOPs seems easier.
Using GOPs seems to simplify things, however once they are loaded, they can not be unloaded
if the actual real class is required. That means that GOPs are strictly forbidden if the real
class is available.
While GOPs can be very useful for class-path independent import/export, they can not be used
during schema evolution because they can not be evolved.
One problem with strict separation of GO/PC: For example during schema evolution, if a GO field
is set through the Handle-API, the value may be an SCO such as an array. But to maintain the
separation of GO/PC, this means that the array and all that it contains(!!!) such as other SCOs
or PCs need to be turned into GOs. Or: we allow references from GO to PC but not the other way
around...
TBD.
Import/export should work fine with pure GenericObjects (+generic SCOs). Import/export should also
keep using GOs/GOPs in order to allow dynamic import of XMLs(?).
Schema evolution is more complicated.

Bad error message/exception when DB is already in use by another process

java.lang.NullPointerException
at org.zoodb.jdo.internal.server.StorageRootFile.close(StorageRootFile.java:104)
at org.zoodb.jdo.internal.server.DiskAccessOneFile.close(DiskAccessOneFile.java:544)
at org.zoodb.jdo.internal.model1p.Node1P.closeConnection(Node1P.java:162)
at org.zoodb.jdo.internal.Session.close(Session.java:461)
at org.zoodb.jdo.PersistenceManagerImpl.close(PersistenceManagerImpl.java:130)
at eu.ideagarden.backend.cloud.Storage.closeDB(Storage.java:137)
at eu.ideagarden.backend.cloud.Storage.close(Storage.java:126)
at eu.ideagarden.backend.cloud.IdGeneratorTest.testDetach(IdGeneratorTest.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

float/double indexing

Implement indexing and query support for float/double primitives

Introduce mutual exclusion for JavaObject vs GenericObject

PersistenceManager.addLifeCycleListener not working for sub-classes

PersistenceManager.addLifeCycleListener() should ensure that events are also thrown for sub-classes of defined classes

ZooClass.dropinstances() may corrupt database

Under certain circumstances (see Test_030_Schema), ZooClass.dropschema() can corrupt the database. The corruption occurs in the attribute index of the class where instances are dropped.

This appears to be related to the AbstractIndex.clear() method being called. It only happens if enough (>100?) instances of a different class are created in the same transaction, and if there has been a commit() before the current transaction. Very weird.

RMI test fails on Linux

The RMI test in Test_020 works fine on Windows but fail on Linux.
Once fixed, re-enable the test in pom.xml

Rename ZooClass/ZooField.remove() to drop(); ZooHandle.remove() to delete()

This is compatible with SQL, which uses 'drop' for tables and 'delete' for instances

Implement query-NOT

Was implemented a whole ago, v0.2?

Improve queries

setOrdering()
methods such as startsWith() / contains()
Object comparison
set result type
aggregators
Internal: merge queries instead of merging the result (indexed OR-queries)

Implement/fix PARAMETERS keyword for queries

Implement/fix PARAMETERS keyword for queries. Parameters generally work, but only via programmatic interface, not via keyword.

Consider using slf4j

Fix chicken-egg schema problem

If two classes A and B reference each other, schema evolution should fail if only one of them is defined or declared through the schema API.

This is already covered in Test_030_Schema.testSchemaCreationChickenEgg().

This is somewhat complicated, because B could be intended to be an SCO. In that case, the current behaviour would be legal.

According to JDO, SCO support is only required for basic Java types (Numbers, String, Collections, ...), however enhanced SCO support is certainly useful for additional classes.
But, SCO support is also inherently dangerous if the SCO class evolves (or becomes an FCO).

Idea: Prevent SCOs by default but show in error message that this behaviour is configurable?
--> Sounds good!

No. We allow SCOs for all classes except:

Classes from org.jdo.*
Classes from org.zoodb.*
Classes that are sub-classes of ZooPCImpl

Make auto-creation of schema the default

JDOUserException "Too few arguments given, parameter count: 1" when String parameter in query is not in quotes

Executing this code

String name = "john";
Query query = pm.newQuery(User.class, "name == " + name);
query.execute();

produces a JDOUserException with the message mentioned in the title. The message could be more descriptive about what to do, since putting the name value in quotes solves the problem:

pm.newQuery(User.class, "name == '" + name + "'");

The User class looks like this:

public class User {

  private String name;

  @SuppressWarnings("unused")
  private User(){
    // hidden default constructor for ZooDB.
  }

  public User(final String name) {
    this.name = name;
  }

  public String getName() {
    this.zooActivateRead();
    return this.name;
  }

  public void setName(final String name) {
    this.zooActivateWrite();
    this.name = name;
  }

}

Queries on indexed string attribute return wrong results if some strings are NULL

See Test_122 (3ab8658).

Implement memory mapped files

OGT throws NPE when serializing objects of type java.lang.Object.

Storing an instance of the following class would throw an NPE in the ObjectGraphtraverser:

class PureObject extends PersistenceCapableImpl {
    private Object o = new Object();

    protected PureObject() {
        // no-argument constructor for zoodb.
    }

    public Object getO() {
        return o;
    }
}

A unit test is available in Test_084_SerailizationBugRefToPM.testSerializationPureObject().