permazen/permazen

Language-Natural Persistence Layer for Java

Permazen is a different persistence layer for Java

Permazen is a ground-up rethink of Java persistence. It has been in production use in commercial software since 2015.

What is it?

Permazen is:

• A Java persistence layer for SQL, key-value, or in-memory databases
• A rigorously defined, modular key/value API with adapters for 15+ database technologies
• An object serialization framework
• An automated schema management framework
• A library for inverting Java references
• A library for precise, non-local field change notifications
• An embeddable Java command line interface (CLI)

Got five minutes? Read Persistence Programming: Are We Doing This Right? in the Jan/Feb 2022 issue of ACM Queue magazine for an overview of the Permazen "story".

What's the motivation?

Persistence is central to most applications. But there are many challenges involved in persistence programming that lie outside of the domain of simply storing the data.

Mainstream Java solutions such as JDBC, JPA and JDO were designed simply to give Java programmers access to existing database functionality. They address the "storage" problem, but leave many other important issues that are inherent to persistence programming poorly addressed, or not addressed at all.

Permazen is a completely different way of looking at persistence programming. Instead of starting from the storage technology side, it starts from the programming language side, asking the simple question, "What are the issues that are inherent to persistence programming, regardless of programming language or database storage technology, and how can they be addressed at the language level in the simplest, most correct, and most language-natural way?"

With Permazen, not only are many issues inherent to persistence programming solved more easily and naturally than before, but also many issues that traditional solutions don't address at all are solved, and some entirely new, useful functionality is added.

Permazen was inspired by years of frustration with existing persistence solutions, in particular JPA. Compared to using JPA, building an application with Permazen is a refreshingly straightforward experience.

Ask these questions of your persistence solution:

• Configuration complexity Do we have to explicitly configure details of how data is mapped? Are we forced to (ab)use the programming language to address what are really database configuration issues?
• Query language concordance Does the code that performs queries look like regular Java code, or do we have to learn a new “query language”?
• Query performance transparency Is the performance of a query visible and obvious from looking at the code that performs it?
• Data type congruence Are database types equivalent to Java types across the entire domain of values? Are we guaranteed to always read back the same value we write?
• First class offline data Can it be precisely defined which data is copied out of a transaction? Does offline data have all the rights and privileges of “online” (i.e., transactional) data? Does this include the ability to query indexes, and a framework for handling schema differences? Can offline data be easily serialized/deserialized?
• Schema verification Is the schema assumed by the code cross-checked against the schema actually present in the database? Are we always guaranteed a consistent interpretation of stored data?
• Incremental schema evolution Can multiple schemas exist at the same time in the database, to support rolling upgrades? Can data be migrated incrementally, i.e., without stopping the world? Are we free from "whole database" migration operations that would limit scalability?
• Structural schema changes Are structural schema updates performed entirely automatically for us?
• Semantic schema changes Is there a convenient way to specify semantic schema updates, at the language level (not the database level)?
• Schema evolution type safety Is type safety and data type congruence guaranteed across arbitrary schema migrations?
• Transactional validation Does validation, including reference validation, occur only at the end of the transaction (as it should), or randomly and inconveniently in the middle?
• Cross-object validation Is it possible to define validation constraints that span multiple objects/records? Can we register for data-level notifications about changes in non-local objects?
• Custom types and indexes Is it possible to define custom data types, have them be indexed? Is it easy to define arbitrary custom indexes?
• Language-level data maintainability Can database maintenance tasks and queries be performed via a command line interface (CLI) using Java types, values, and expressions (including Java 8 streams and lambdas)? Are there convenient tools for manual and scripted use?
• Data store independence Are we restricted to using only a specific type of database technology, or can virtually any database technology be used by implementing a simple API, making it easy to change later if needed?

Permazen attempts to address all of these issues. It does so by treating the database as just a sorted key/value store and implementing everything else in Java:

• Encoding/decoding of field values
• Referential integrity; forward/reverse cascades
• Indexes (simple and composite)
• Query views
• Schema management
• Change notification
• Validation queues
• "Offline" data
• Command line interface
• GUI database editor

Permazen also adds several new features that traditional databases don't provide. For example, you can define your own basic types and then index them however you want.

What's the Downside?

Permazen redefines the "line of demarcation" between a Java application and its supporting database. However, this has some implications:

For an equivalent query, Permazen will perform more frequent, but smaller, database accesses. As a result, in situations where the code and the data are separated by a high latency network, Permazen will probably be too slow. Permazen is best suited for scenarios where the code and the data are running on the same machine or in close proximity.

You have to learn something new. Persistence programming with Permazen requires a different way of thinking. For example, a "DAO" layer is often no longer necessary, and you may have to think harder about how to query your data efficiently, instead of crossing your fingers and hoping the database figures it out for you, because there is no query planner (you are the query planner).

More flexible type hiearchies are possible, but it's also easy to make a mess. JPA has support for class inheritance and only partial support for generics. Permazen supports interface inheritance (including Java's equivalent of "mix-ins") and fully supports generic types. The restrictions imposed by JPA tend to force model classes to stay simpler. With Permazen, implementing an interface (directly or indirectly) can mean that a model class inherits a bunch of new persistent fields.

What are some interesting Java classes to look at?

Key/Value Layer

• KVStore - A thing that contains key/value pairs
• KVDatabase - A thing that persists key/value pairs
• KVTransaction - A transaction for a KVDatabase
• RaftKVDatabase - A distributed KVDatabase based on the Raft consensus algorithm.

Java Layer

• Permazen - A Permazen database instance
• PermazenTransaction - A Permazen database transaction
• FieldType - How all simple database types are defined
• PermazenObject - Interface implemented by runtime-generated concrete model classes
• @PermazenType - Annotation identifying your database classes
• @PermazenField - Annotation configuring your database fields

Other

• @OnChange - How change notifications are delivered
• ReferencePath - Describes a path between objects that hops through one or more forward and/or inverse references
• @OnVersionChange - How schema update "fixups" are defined
• PermazenObjectHttpMessageConverter - For sending/receiving versioned graphs of objects over the network using Spring

How are Java object, data structures, and indexes mapped into key/value pairs?

See LAYOUT.md.

Permazen Article

Persistence Programming: Are We Doing This Right? appears in the Jan/Feb 2022 issue of ACM Queue magazine. This gives a good overview of how Permazen tries to improve the persistence programming experience.

Permazen Slides

For a quick overview, check out these slides from a JSimpleDB talk at a local Java user's group (Permazen was previously named JSimpleDB).

Permazen Paper

For a deeper understanding of the motivation and design decisions behind Permazen, read Permazen: Language-Driven Persistence for Java.

Abstract:

Most software applications require durable persistence of data. From a programmer’s point of view, persistence has its own set of inherent issues, e.g., how to manage schema changes, yet such issues are rarely addressed in the programming language itself. Instead, how we program for persistence has traditionally been driven by the storage technology side, resulting in incomplete and/or technology-specific support for managing those issues.

In Java, the mainstream solution for basic persistence is the Java Persistence API (JPA). While popular, it also measures poorly on how well it addresses many of these inherent issues. We identify several examples, and generalize them into criteria for evaluating how well any solution serves the programmer’s persistence needs, in any language. We introduce Permazen, a persistence layer for ordered key/value stores that, by integrating the data encoding, query, and indexing functions, provides a more complete, type-safe, and language-driven framework for managing persistence in Java, and addresses all of the issues we identify.

Installing Permazen

Permazen is available from Maven Central:

    <dependency>
        <groupId>io.permazen</groupId>
        <artifactId>permazen-main</artifactId>
    </dependency>

You should also add the key/value store module(s) for whatever key/value store(s) you want to use, e.g.:

    <dependency>
        <groupId>io.permazen</groupId>
        <artifactId>permazen-kv-sqlite</artifactId>
    </dependency>

There is a demo distribution ZIP file that lets you play with the Permazen command line and GUI, using a simple database of the solar system.

Documentation

Documentation and links:

• Introduction
• Getting Started
• FAQ
• API Javadocs
• Bullet-point JPA Comparison
• Key/value pair database layout
• Permazen Users discussion group
• Auto-generated Maven Site

permazen's Issues

Playing well with other libraries like Jackson for serialization

For modern web frameworks, JSON can be made automatically for return values from method invocation that are bound to HTTP-GET operations. For example List<Person> can be turned into a list a people in JSON.

Strategies are needed for excluding the fields from the JSimpleDB generated subclass of Person though. If the serialization tech is Jackson, there are a number of strategies but each is quite a bit of obscure code. Especially if the web-framework in question instantiates its own ObjectMapper.

Jackson allows @JsonIgnore to prefix any field that should not be serialized. Java is quite generous for annotations that are missing from the classpath at runtime so adding that to every @JsimpleDB generated field would work.

Alternatively (and best of all) making each @JsimpleDB generated field transient would work too. This one would work for Gson too.

Thoughts?

Sample code out of date with release

JTransaction in the sample code has 3 extra get methods not available in the 1.3.5 release version.
get(ObjId), get(ObjId, Class), get(T)

This is a good reason to separate the demo/sample code into separate projects from the core code.

Delete cascade combined with DeleteAction.EXCEPTION fails on delete of referring object

If you have this:

@JSimpleClass
public class Person {
    @JField(cascadeDelete = true)
    public abstract Person getFriend();
    public abstract void setFriend(Person friend);
}

and you:

Person a = jtx.create(Person.class);
Person b = jtx.create(Person.class);
a.setFriend(b);

and then you

a.delete()

you get

ReferencedObjectException: object a cannot be deleted
  because it is still referenced by field X in object b,
  which is configured for error on deletion

The DeleteAction.EXCEPTION check should first check whether the referring object is being deleted.

permazen.io website is hosed

It's not being answered by github.com correctly so we get an invalid certificate error.

MVStore Support?

Permazen supports a great many KVs, but not the very popular MVStore (https://www.h2database.com/html/mvstore.html). Is there a reason for this?

The reason I ask is because we are interested in a kind of mixed-storage model. A single MVStore database that would contain both relational data (and indexes) as well as XML documents. So you might have Permazen talking to one MVStore Map and an existing XMLDocumentStore talking to a different MVStore Map. Queries could be done using Permazen and its nice support for indexes but data would be stored using XML. We already have code that works with XML documents and MVStore which is why I ask about MVStore in particular. But we're not wedded to MVStore...

This is part of a larger problem: "pass through data." We want to store data (as XML) which is not actually part of our object model. I wonder if Permazen has any support for this? I suppose it would be possible to define a field of type byte[] right?

Add support for meta-data striping

Suggested by Scott Ziegler.

One of FoundationDB's limitations is "limited read load balancing"; see

  https://foundationdb.com/documentation/known-limitations.html#limited-read-load-balancing

Since every Transaction requires reading the transaction meta-data, those keys 
could become a bottleneck.

To handle this, add support for meta-data striping. The idea is that a Database 
could be configured with a list of storage ID's that are dedicated to meta-data 
striping. These would be unavailable for use with schema types and fields.

Then each transaction would read the meta-data from a randomly chosen stripe.

Issues to deal with:
* Stripe zero (what we have today) is always included; this allows the striping 
to be inferred.
* The list of meta-data stripe storage ID's will have to be contained in the 
meta-data, hence there is a chicken/egg situation. Include checks so that 
meta-data read from a non-zero stripe is verified; throw exception if recorded 
striping does not match configured striping, where "match" means configured 
striping includes a storage ID not in the recorded striping. So you can add 
stripes while the system is running.
* Obviously trying to use a schema whose storage IDs conflict with the 
database's striping throws an exception. 
* Add CLI command that adds or removes stripes to/from an already populated 
Database. Removing a stripe may cause running applications to start failing.

Original issue reported on code.google.com by [email protected] on 16 May 2014 at 1:33

Auto-boxing equivalent required for Kotlin compatibility

Kotlin unifies int and java.lang.Integer at the language level. Likewise for other primitives. This confuses JSimpleDB and yields errors like this one:

Exception in thread "main" java.lang.IllegalArgumentException: invalid value type int for index query on field `age' in class net.plan99.jsimdb.Person: should be a super-type of java.lang.Integer

given this sort of code:

fun queryByAge() = dbtx.queryIndex(Person::class.java, "age", Int::class.java).asMap()

It works if I change it to:

fun queryByAge() = dbtx.queryIndex(Person::class.java, "age", Integer::class.java).asMap()

but this triggers Kotlin warnings and really, it'd be nicer if int and Integer were treated as equivalent.

Infinite Loop with 'Hello World' code in SQLKVDatabase

First I've got to say, this looks like a really awesome project!

I'm hoping to use it in a new project, but for a database I'm required to use MS SQL 2012, and I encountered an infinite loop.

To get started, I pulled the required jars for jsimpledb-kv-3.4.0, and I fetched the latest JTDS/MS drivers. I then followed the instructions at the getting started page. I first ran everything with the XMLKVDatabase and had no issues.

I then extended the SQLKVDatabase to override createPutStatement() with a transaction. Writing works, but during the read I ran into the loop at getFriendsToMe().

I overrode createGetRangeStatement() and changed the range from column>=? to column>?, and then received the expected output without any issue.

Do you believe this problem is specific to MS SQL? And is there a test procedure I should use to ensure interoperability with the database?

int.class and Integer.TYPE in txn.queryIndex() statements (etc)

java.lang.IllegalArgumentException: invalid value type int for index query on field `id' in class de.protubero.ajs.Person: should be a super-type of java.lang.Integer

I was trying to work through some issues and I flipped Integer.class to Integer.TYPE.

txn.queryIndex(Person.class, "id", Integer.TYPE);
// vs
txn.queryIndex(Person.class, "id", Integer.class);

Exception messages could be more helpful (add 'do not use .TYPE' words of advice etc), or the wiki documentation could contain the same information in the broad area of primitives, boxing etc.

Introduction doesn't mention JTransaction.setCurrent()

Seems odd to me as getCurrent() throws and exception if CURRENT not set.

Also, that exception isn't talked of in the introduction.

And I'd personally like to be able to use JSimpleDB without thread-local being involved. i.e. Instantiate it myself.

Why not non-final classes?

In the introduction, "The model class can be an abstract class or an interface" is stated.

Why not support POJOs that are not final and that do not have any final methods?

Incidentally, a later bullet: "This makes life easier, but there is no requirement that the model class extend or implement anything" should be indented.

Weirdness for POJO field access

Here's my POJO with annotations added for JSimpleDB:

@JSimpleClass
public class Person {

	private int id;
	private String name;
	private String email;
	private int age;

	@JField(indexed = true)
	public int getId() {
		return id;
	}
	public void setId(int id) {
		this.id = id;
	}

	@JField
	public String getName() {
		return name;
	}
	public void setName(String name) {
		this.name = name;
	}

	@JField
	public String getEmail() {
		return email;
	}
	public void setEmail(String email) {
		this.email = email;
	}

	@JField
	public int getAge() {
		return age;
	}
	public void setAge(int age) {
		this.age = age;
	}

	public void updateWith(Person upToDatePerson) {
		this.age = upToDatePerson.age;
		this.email = upToDatePerson.email;
		this.name = upToDatePerson.name;
	}

//	@Override
//	public String toString() {
//		return "Person{" +
//				"id=" + id +
//				", name='" + name + '\'' +
//				", email='" + email + '\'' +
//				", age=" + age +
//				'}';
//	}

	@Override
	public String toString() {
		return "Person{" +
				"id=" + getId() +
				", name='" + getName() + '\'' +
				", email='" + getEmail() + '\'' +
				", age=" + getAge() +
				'}';
	}
}

There's two alternate toString() implementations there. One that Intellij made for me that uses fields to compose the RV. In use those were 0 and null despite prior setXyz() calls on the instance. When I changed it to use the getters it worked as expected. I deduce that the POJOs own fields are effectively disabled. While I understand that JSimpleDB wants to subclass POJOs, I feel it could be more faithful to the definition of POJOs if each generated method were to also invoke the super with the same args.

Or give up on POJOs and suggest abstractions only.

The trouble with the latter is that web frameworks like Spring want to create a temp 'model' object from request params (and cookie vars and form fields) to hand to a controller method, and according to MVC* that model object could be discarded and not at all saved - subject to the control flow within the controller method (and whatever collaborating methods it calls). Given 'not save' is a viable lifecycle for a Person instance (in this case) that'd be coming through a web-UI, then we cannot insist that all Person instances are made via JTransaction.create(modelClass). If we did it would be data-binding*.

Say we did insist that. I'd end up with Person.class for saving to JSimpleDB, and PersonUI.class with all the same fields and methods for the UI framework to casually create at will, and a checklist activity for the dev team to ensure that the two Person classes tracked each other's fields etc over the months and years. Or a code gen situation.

* https://paulhammant.com/2015/04/29/mvc-misunderstood-for-37-years - find "Late to MVC Party" in page

small typo on wiki

Hi - on [1] there is a small typo: look for "consruct", this should be "construct".

Cheers, Oscar

[1] https://github.com/archiecobbs/jsimpledb/wiki/Introduction

A POJO test case - add this to BasicTest.java if you want

    @Test
    @SuppressWarnings("unchecked")
    public void testBasicPojoStuff() throws Exception {
        final JSimpleDB jdb = BasicTest.getJSimpleDB();

        JTransaction txn = jdb.createTransaction(true, ValidationMode.MANUAL);
        PojoPerson p = txn.create(PojoPerson.class);
        p.setId(123);
        p.setName("Fred");
        PojoPerson p2 = keyedOnId(txn).asMap().get(123).first();
        assertEquals(p2.getId(), 123);
        assertEquals(p2.getName(), "Fred");
        NavigableSet<PojoPerson> ppl = keyedOnId(txn).asMap().get(456);
        assertNull(ppl);
        txn.commit();

        try {
            keyedOnId(txn);
            fail("should have barfed, even read-only operations have to be in a txn");
        } catch (StaleTransactionException e) {
            // expected;
        }

        txn = jdb.createTransaction(true, ValidationMode.MANUAL);
        p2 = keyedOnId(txn).asMap().get(123).first();
        assertEquals(p2.getName(), "Fred");
        txn.rollback();

        txn = jdb.createTransaction(true, ValidationMode.MANUAL);
        p2 = keyedOnId(txn).asMap().get(123).first();
        txn.delete((JObject) p2);

        assertFalse(keyedOnId(txn).asMap().containsKey(123));

        txn.commit();

    }

    private Index<Integer, PojoPerson> keyedOnId(JTransaction txn) {
        return txn.queryIndex(PojoPerson.class, "id", Integer.class);
    }

    @JSimpleClass()
    public static class PojoPerson {
        private int id;
        private String name;
        @JField(indexed = true)
        public int getId() {
            return id;
        }
        public void setId(int value) {
            id = value;
        }
        @JField()
        public String getName() {
            return name;
        }
        public void setName(String value) {
            name = value;
        }
    }

Verify SQL Server properly compares binary values

Does SQL Server consider 0x1234 equal to 0x123400 ?

This is of course incompatible with JSimpleDB's assumptions about key/value stores and would mean MSSQLKVDatabase is broken, even though it appears to pass the unit tests OK (in my tests).

Some worrisome data points:

http://stackoverflow.com/questions/26255195/how-does-varbinary-data-sorted/26257328?noredirect=1#comment74644636_26257328

https://dba.stackexchange.com/questions/48660/comparing-binary-0x-and-0x00-turns-out-to-be-equal-on-sql-server

Should not include log4j slf4j adapter

The point of SLF4J is that the user provides their own adapter and libraries just depend on the facade, right?

"IllegalArgumentException: invalid target type" exception during commit

Caused by: java.lang.IllegalArgumentException: invalid target type com.example.HasRealtimeLocation for index query on field `realtimeLocation' in interface com.example.HasRealtimeLocation: should be a super-type or sub-type of com.example.AbstractGridObject
    at org.jsimpledb.IndexInfo$ValueCheck.checkAndGetKeyRanges(IndexInfo.java:240) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.IndexInfo.<init>(IndexInfo.java:92) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.IndexInfo.<init>(IndexInfo.java:41) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.IndexInfoKey.getIndexInfo(IndexInfoKey.java:33) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.JSimpleDB$1.load(JSimpleDB.java:144) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.JSimpleDB$1.load(JSimpleDB.java:141) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3542) ~[guava-19.0-guava.jar:?]
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2323) ~[guava-19.0-guava.jar:?]
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2286) ~[guava-19.0-guava.jar:?]
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-19.0-guava.jar:?]
    at com.google.common.cache.LocalCache.get(LocalCache.java:3953) ~[guava-19.0-guava.jar:?]
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3957) ~[guava-19.0-guava.jar:?]
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4875) ~[guava-19.0-guava.jar:?]
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4881) ~[guava-19.0-guava.jar:?]
    at org.jsimpledb.JSimpleDB.getIndexInfo(JSimpleDB.java:871) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.JTransaction.doValidate(JTransaction.java:1645) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.JTransaction.access$400(JTransaction.java:188) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.JTransaction$1.run(JTransaction.java:1552) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.JTransaction.performAction(JTransaction.java:1572) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.JTransaction.validate(JTransaction.java:1549) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]
    at org.jsimpledb.JTransaction.commit(JTransaction.java:1488) ~[jsimpledb-2.4.0-jsimpledb-main.jar:2.4.0]

aspectj-maven-plugin has changed coordinates

New coordinates are dev.aspectj:aspectj-maven-plugin:1.13

https://package-search.jetbrains.com/package?id=dev.aspectj%3Aaspectj-maven-plugin

Currently the POM does not resolve.

CVE in dependency org.glassfish:javax.el:3.0.1-b11

When adding io.permazen:permazen-main:4.1.9 to a java project in Jetbrains IntelliJ, I get a warning about CVE-2021-28170 within transitive dependency org.glassfish:javax.el:3.0.1-b11 which was fixed in 3.0.4 or newer.
As transition from javax to jakartee happened right before 3.0.2 release, this dependendency has to be updated (and dependeing code to be refactored) to org.glassfish:jakarta.el:3.0.4 (or newer)

Generated subclasses could implement toString using ObjDumper automatically

The suggested toString implementation in the best practices doc seems like it could be generated by the framework automatically.

How to get started with BDB

BerkeleyKVDatabase kvdb = new BerkeleyKVDatabase();
kvdb.setDirectory(new File("."));
kvdb.setDatabaseName("fred");
kvdb.start();
jdb = new JSimpleDBFactory()
    .setDatabase(new Database(kvdb))
    .setSchemaVersion(-1)
    .setModelClasses(Person.class).newJSimpleDB();
final JTransaction txn = jdb.createTransaction(true, ValidationMode.AUTOMATIC);
JTransaction.setCurrent(txn);

The above gives:

org.jsimpledb.kv.RetryTransactionException: transaction must be retried

	at org.jsimpledb.kv.bdb.BerkeleyKVTransaction.wrapException(BerkeleyKVTransaction.java:265)
	at org.jsimpledb.kv.bdb.BerkeleyKVTransaction$CursorIterator.findNext(BerkeleyKVTransaction.java:395)
	at org.jsimpledb.kv.bdb.BerkeleyKVTransaction$CursorIterator.hasNext(BerkeleyKVTransaction.java:312)
	at org.jsimpledb.core.Database.verifySchemas(Database.java:471)
	at org.jsimpledb.core.Database.createTransaction(Database.java:304)
	at org.jsimpledb.core.Database.createTransaction(Database.java:254)
	at org.jsimpledb.JSimpleDB.createTransaction(JSimpleDB.java:581)
	at org.jsimpledb.JSimpleDB.createTransaction(JSimpleDB.java:551)
	at de.protubero.ajs.PersonStoreImpl.selectById(PersonStoreImpl.java:80)
	at de.protubero.ajs.PersonStoreTest.shouldBeAbleToDeleteKnownPerson(PersonStoreTest.java:33)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: com.sleepycat.je.LockTimeoutException: (JE 6.4.9) Lock expired. Locker 1328238652 11_main_Txn: waited for lock on database=fred LockAddr:785570251 LSN=0x0/0x1265 type=RANGE_READ grant=WAIT_NEW timeoutMillis=500 startTime=1502933461314 endTime=1502933461816
Owners: [<LockInfo locker="1926096844 10_main_Txn" type="RANGE_WRITE"/>]
Waiters: []

	at com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:736)
	at com.sleepycat.je.txn.LockManager.makeTimeoutException(LockManager.java:590)
	at com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:417)
	at com.sleepycat.je.txn.LockManager.lock(LockManager.java:295)
	at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:567)
	at com.sleepycat.je.txn.Locker.lock(Locker.java:504)
	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:3551)
	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:3346)
	at com.sleepycat.je.dbi.CursorImpl.lockLNAndCheckDeleted(CursorImpl.java:2149)
	at com.sleepycat.je.dbi.CursorImpl.lockAndGetCurrent(CursorImpl.java:2074)
	at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:2552)
	at com.sleepycat.je.Cursor.retrieveNextCheckForInsertion(Cursor.java:3212)
	at com.sleepycat.je.Cursor.retrieveNextNoDups(Cursor.java:3094)
	at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:2836)
	at com.sleepycat.je.Cursor.getNext(Cursor.java:1117)
	at org.jsimpledb.kv.bdb.BerkeleyKVTransaction$CursorIterator.findNext(BerkeleyKVTransaction.java:379)
	... 30 more

If I comment out the start() operation:

java.lang.IllegalStateException: not started

	at com.google.common.base.Preconditions.checkState(Preconditions.java:444)
	at org.jsimpledb.kv.bdb.BerkeleyKVDatabase.createTransaction(BerkeleyKVDatabase.java:274)
	at org.jsimpledb.kv.bdb.BerkeleyKVDatabase.createTransaction(BerkeleyKVDatabase.java:267)
	at org.jsimpledb.kv.bdb.BerkeleyKVDatabase.createTransaction(BerkeleyKVDatabase.java:41)
	at org.jsimpledb.core.Database.createTransaction(Database.java:251)
	at org.jsimpledb.JSimpleDB.createTransaction(JSimpleDB.java:581)
	at org.jsimpledb.JSimpleDB.createTransaction(JSimpleDB.java:551)

So I can't see an example of how to programmatically start BDB via JSimpleDB, and don't know what to do next to get it going. What am I doing wrong?

Oh, and it would be cool to have fluent interfaces like withDirectory() and withDatabaseName() for in-line fun:

jdb = new JSimpleDBFactory()
    .setDatabase(new Database(new BerkeleyKVDatabase()
        .withDirectory(new File(".")).withDatabaseName("fred")))
    .setSchemaVersion(-1)
    .setModelClasses(Person.class).newJSimpleDB();

TransactionTimeoutException while debugging in IDE is going to be very problematic for adoption

First I had one problem: something isn't working as I expect it to. Could be my code, but who knows, so I'll crank up the debugger in Intellij.

In the IDE I see: org.jsimpledb.kv.TransactionTimeoutException: transaction taking too long: hold timeout of 5000ms has expired

Now I have two problems :-(

Also: timeouts are not mentioned in the Introduction wiki page.

https://tomp2p.net/ support

can it support Distributed Hash Table like Tomp2p DB?

Read all meta-data in a single range read

When a new Transaction is opened, the meta-data (encoding key and recorded 
schemas) is read via two reads: a get() for the encoding key and a getRange() 
for the schemas).

Optimize this slightly by reading both encoding key and schemas with a single 
getRange() call.

Original issue reported on code.google.com by [email protected] on 16 May 2014 at 1:39

How does one efficiently perform a join?

I don't see a way to do it without a N+1 query which is pretty much the opposite of efficient.

-- PG

Unit test failure in XMLKVDatabase

 INFO: [main] starting testParallelTransactions() on [org.jsimpledb.kv.simple.XMLKVDatabase@1c9520e4]
 INFO: [main] starting testParallelTransactions() iteration 0
DEBUG: [Random[0]] *** Random[0] STARTING
DEBUG: [Random[0]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@5f5ec5f0
DEBUG: [Random[1]] *** Random[1] STARTING
DEBUG: [Random[2]] *** Random[2] STARTING
DEBUG: [Random[1]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@15419025
DEBUG: [Random[3]] *** Random[3] STARTING
DEBUG: [Random[4]] *** Random[4] STARTING
DEBUG: [Random[2]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@2f1ada40
DEBUG: [Random[3]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@656c281d
DEBUG: [Random[5]] *** Random[5] STARTING
DEBUG: [Random[6]] *** Random[6] STARTING
DEBUG: [Random[6]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@692ef3d8
DEBUG: [Random[7]] *** Random[7] STARTING
DEBUG: [Random[8]] *** Random[8] STARTING
DEBUG: [Random[5]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@59b5a0b
DEBUG: [Random[9]] *** Random[9] STARTING
DEBUG: [Random[4]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@5e0e9cd0
DEBUG: [Random[11]] *** Random[11] STARTING
DEBUG: [Random[10]] *** Random[10] STARTING
DEBUG: [Random[9]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@6b8099d3
DEBUG: [Random[12]] *** Random[12] STARTING
DEBUG: [Random[13]] *** Random[13] STARTING
DEBUG: [Random[8]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@6bcc7462
DEBUG: [Random[14]] *** Random[14] STARTING
DEBUG: [Random[7]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@73bace2b
DEBUG: [Random[14]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@5cbcdfdb
DEBUG: [Random[15]] *** Random[15] STARTING
DEBUG: [Random[16]] *** Random[16] STARTING
DEBUG: [Random[17]] *** Random[17] STARTING
DEBUG: [Random[15]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@3720a5d7
DEBUG: [Random[18]] *** Random[18] STARTING
DEBUG: [Random[13]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@64492e19
DEBUG: [Random[19]] *** Random[19] STARTING
DEBUG: [Random[18]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@4ca6a9d0
 WARN: [Finalizer] org.jsimpledb.kv.simple.SimpleKVTransaction@10f8ae75 leaked without commit() or rollback()
DEBUG: [Random[12]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@36858c07
DEBUG: [Random[20]] *** Random[20] STARTING
DEBUG: [Random[10]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@1a785a79
DEBUG: [Random[21]] *** Random[21] STARTING
DEBUG: [Random[11]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@6d4d3c3b
DEBUG: [Random[23]] *** Random[23] STARTING
DEBUG: [Random[22]] *** Random[22] STARTING
DEBUG: [Random[24]] *** Random[24] STARTING
DEBUG: [Random[21]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@1ef20bb6
DEBUG: [Random[24]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@26ba4eb8
DEBUG: [Random[20]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@118625e5
DEBUG: [Random[19]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@696103e5
DEBUG: [Random[17]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@6046e56f
DEBUG: [Random[16]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@4e810bca
DEBUG: [Random[22]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@30726eca
DEBUG: [Random[23]] *** CREATED TX org.jsimpledb.kv.simple.XMLKVTransaction@28c7a89f
DEBUG: [Random[0]] *** COMMITTING TX org.jsimpledb.kv.simple.XMLKVTransaction@5f5ec5f0
DEBUG: [Random[0]] *** COMMITTED TX org.jsimpledb.kv.simple.XMLKVTransaction@5f5ec5f0
DEBUG: [Random[1]] *** Random[1] FINISHED
DEBUG: [Random[6]] *** Random[6] FINISHED
DEBUG: [Random[1]] *** BEGIN Random[1] LOG ***

seed = 7616751421776965593
getAtMost: 5e -> null
new values:
  knowns={}
  puts=[]
  emptys=[[,5e)]
failed: java.lang.AssertionError: Random[1]: tx contains {05,c7c9ea7c74f2869d} but
  knowns={}
  puts=[]
  emptys=[[,5e)]
  tx={05=c7c9ea7c74f2869d, 0d=abd31c9485d86523, 14=7a, 1c=45, 80=02, e2=c7}
java.lang.AssertionError: Random[1]: tx contains {05,c7c9ea7c74f2869d} but
  knowns={}
  puts=[]
  emptys=[[,5e)]
  tx={05=c7c9ea7c74f2869d, 0d=abd31c9485d86523, 14=7a, 1c=45, 80=02, e2=c7}
    at org.jsimpledb.kv.test.KVDatabaseTest$RandomTask.test(KVDatabaseTest.java:792)
    at org.jsimpledb.kv.test.KVDatabaseTest$RandomTask.run(KVDatabaseTest.java:564)


*** END Random[1] LOG ***
DEBUG: [Random[6]] *** BEGIN Random[6] LOG ***

seed = 20306850681500960
getAtMost: 3b -> null
new values:
  knowns={}
  puts=[]
  emptys=[[,3b)]
failed: java.lang.AssertionError: Random[6]: tx contains {05,c7c9ea7c74f2869d} but
  knowns={}
  puts=[]
  emptys=[[,3b)]
  tx={05=c7c9ea7c74f2869d, 0d=abd31c9485d86523, 14=7a, 1c=45, 80=02, e2=c7}
java.lang.AssertionError: Random[6]: tx contains {05,c7c9ea7c74f2869d} but
  knowns={}
  puts=[]
  emptys=[[,3b)]
  tx={05=c7c9ea7c74f2869d, 0d=abd31c9485d86523, 14=7a, 1c=45, 80=02, e2=c7}
    at org.jsimpledb.kv.test.KVDatabaseTest$RandomTask.test(KVDatabaseTest.java:792)
    at org.jsimpledb.kv.test.KVDatabaseTest$RandomTask.run(KVDatabaseTest.java:564)


*** END Random[6] LOG ***

What if I don't commit and don't rollback a txn ?

It is not clear what is going to happen to pending changes if .rollback() isn't called on a txn where changes had been made. Similarly, should you call .rollback() when no changes were made at all, and you're wanting to close the transaction rather than just surrender it to GC.

https://github.com/archiecobbs/jsimpledb/search?q=rollback&type=Wikis&utf8=%E2%9C%93

Vaadin GUI Update

This isn't an issue per say, but after setting up the GUI editor for a schema I wrote, I realized I was no longer able to edit objects with a Set, since the JSet/JMap/JList field builders in JObjectEditorWindow.java are still a TODO and result in an "Invalid value(s)" message.

Though technically not needed, I'd really love to use the editor. Any recommendation to get past this in the meantime? Alternatively, I'm thinking I could hold myself over by writing a very simple implementation of the field builders (a text field of json to represent the set/map/list of either internal IDs or simple data)

AccessDeniedException on AtomicArrayKVStore initialization.

Whether you can open a directory as a regular file is highly platform specific. The following code fails on Windows.

Offending code:

if(!this.directory.isDirectory()) {
    throw new ArrayKVException("file `" + this.directory + "\' is not a directory");
}
this.directoryChannel = FileChannel.open(this.directory.toPath(), new OpenOption[0]);

Stacktrace:
Caused by: java.nio.file.AccessDeniedException: C:\x\y\z\kvstore
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.jsimpledb.kv.array.AtomicArrayKVStore.start(AtomicArrayKVStore.java:353)
... 72 more

ConvertedNavigableMap.get() can return null

^ I don't know why the thing thinks there is nothing in the map. I'll try to debug to a solution for that separately.

However, this method returning null is at odds with example usages from the documentation. Like so:

AtomicUpdateFileOutputStream does not work on Windows

On Windows, need to avoid using it.

Cohesive examples needed and information architecture.

Examples

Unit tests don't give enough info for the end-user to get up and running with. Consider BDB. There should perhaps be a README in here - https://github.com/archiecobbs/jsimpledb/tree/master/jsimpledb-kv-bdb. From that links to example projects.

Implementations

Note too that https://github.com/archiecobbs/jsimpledb (readme) does not mention Berkeley. First linked wiki page Information does, but that links to JavaDocs. One third of the dev world is happy with ref docs, one third with tutorials and one third cohesive examples (like me). Note that StackOverflow is Mecca for example-code people (I have your JSimpleDB answers an upvote).

The unit test for the bdb implementation doesn't yield any larger "how to" information. Nor does the CLI tool.

By contrast, the Jooby project provides a number of cohesive example applications, complete with build unit test and integration tests. Like this one - https://github.com/jooby-project/greeting. Easy to use as a starting point for your own app.

Information rift:

• the Introduction wiki page (listed first) shows snippets of key and value interop.
• the getting started wiki page (listed second) shows wrapping JDB setup and try/finally stuff that must wrap all snippets. Cruicial, IMO, is the usage of .newJSimpleDB() but there's only two refs to that in that page, and not at the top. Note also, you've two orphaned </code> markup vestiges in that doc to excise.
• There should be a wiki page that lists all the implementations of JSimpleDB, including the BDB one. It should be listed in the readme, and linked to again in Introduction and Getting Started. That should include the differentiating setup per implementation.
• .start() and .stop are not really discussed anywhere. Why? What if I don't stop the DB before shutdown? Why can't it start itself on first use?

Don't be put off - I think your stuff is great - and my nooooob experience report should be useful to you.

ValidationException

On creation of a trivial JSimpleDB, new JSimpleDB(persistClasses);

I get the following exception

Exception in thread "Thread-0" javax.validation.ValidationException: Unable to create a Configuration, because no Bean Validation provider could be found. Add a provider like Hibernate Validator (RI) to your classpath.
at javax.validation.Validation$GenericBootstrapImpl.configure(Validation.java:271)
at javax.validation.Validation.buildDefaultValidatorFactory(Validation.java:110)
at org.jsimpledb.JSimpleDB.(JSimpleDB.java:129)
at org.jsimpledb.JSimpleDB.(JSimpleDB.java:161)

A validator class had to be added for JSimpleDB to be created.

<dependency>
    <groupId>org.hibernate</groupId>
    <artifactId>hibernate-validator</artifactId>
    <version>5.2.4.Final</version>
</dependency>   

Options:

• Pull the default ValidatorFactory, and use a null pattern
• Check and verifies a ValidatorFactory and throw a descriptive message and how to fix the problem.
• Provide a hibernate-validator inside the code base, and allow the developer to override.

Convert build from ant to Maven

Maven pom's are an ide agnostic way of sharing project.

The maven pom is unusable jsimpledb/src/mvn/pom.xml

The pom.xml can not be imported into an ide.