Giter Site home page Giter Site logo

nitrite / nitrite-java Goto Github PK

View Code? Open in Web Editor NEW
803.0 27.0 90.0 5.08 MB

NoSQL embedded document store for Java

Home Page: https://bit.ly/no2db

License: Apache License 2.0

Java 97.29% Kotlin 2.71%
nosql embedded-database documentdb android mobile-database database object-storage object-database

nitrite-java's Introduction

Nitrite Database

Build CodeQL codecov javadoc Discussion

nitrite logo

NOsql Object (NO2 a.k.a Nitrite) database is an open source nosql embedded document store. It supports both in-memory and file based persistent store.

Nitrite is an embedded database ideal for desktop, mobile or small web applications.

It features:

  • Embedded, serverless
  • Simple API
  • Document-oriented
  • Schemaless document collection and object repository
  • Extensible storage engines - mvstore, rocksdb
  • Indexing and full-text search
  • Simple query api
  • In-memory and file-based store
  • Transaction support
  • Schema migration support
  • Encryption support
  • Android compatibility (API Level 24)

Kotlin Extension

Nitrite has a kotlin extension called Potassium Nitrite for kotlin developers. Visit here for more details.

Flutter Version

If you are looking for Nitrite for Flutter/Dart, head over to nitrite-flutter.

Deprecation Notice

Nitrite DataGate and Nitrite Explorer is now deprecated and no longer maintained.

Getting Started with Nitrite

NOTE: There are breaking api changes in version 4.x. So please read the guide before upgrading from 3.x.x.

How To Install

To use Nitrite in any Java application, first add the nitrite bill of materials, then add required dependencies:

Maven

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.dizitart</groupId>
            <artifactId>nitrite-bom</artifactId>
            <version>[latest version]</version>
            <scope>import</scope>
            <type>pom</type>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.dizitart</groupId>
        <artifactId>nitrite</artifactId>
    </dependency>

    <dependency>
        <groupId>org.dizitart</groupId>
        <artifactId>nitrite-mvstore-adapter</artifactId>
    </dependency>
</dependencies>

Gradle

implementation(platform("org.dizitart:nitrite-bom:[latest version]"))
    
implementation 'org.dizitart:nitrite'
implementation 'org.dizitart:nitrite-mvstore-adapter'

Examples

A Todo android application is available here to demonstrate the usage of Nitrite in android.

Quick Examples

Initialize Database

// create a mvstore backed storage module
MVStoreModule storeModule = MVStoreModule.withConfig()
    .filePath("/tmp/test.db") 
    .compress(true)
    .build();

// or a rocksdb based storage module
RocksDBModule storeModule = RocksDBModule.withConfig()
    .filePath("/tmp/test.db")
    .build();


// initialization using builder
Nitrite db = Nitrite.builder()
        .loadModule(storeModule)
        .loadModule(new JacksonMapperModule())  // optional
        .openOrCreate("user", "password");

Create a Collection

// Create a Nitrite Collection
NitriteCollection collection = db.getCollection("test");

// Create an Object Repository
ObjectRepository<Employee> repository = db.getRepository(Employee.class);

Annotations for POJO

@Entity(value = "retired-employee",     // entity name (optional), 
    indices = {
        @Index(value = "firstName", type = IndexType.NON_UNIQUE),
        @Index(value = "lastName", type = IndexType.NON_UNIQUE),
        @Index(value = "note", type = IndexType.FULL_TEXT),
})
public class Employee implements Serializable {
    // provides id field to uniquely identify an object inside an ObjectRepository
    @Id
    private long empId;
    private Date joinDate;
    private String firstName;
    private String lastName;
    private String note;

    // ... public getters and setters
}

CRUD Operations

// create a document to populate data
Document doc = Document.createDocument("firstName", "John")
     .put("lastName", "Doe")
     .put("birthDay", new Date())
     .put("data", new byte[] {1, 2, 3})
     .put("fruits", new ArrayList<String>() {{ add("apple"); add("orange"); add("banana"); }})
     .put("note", "a quick brown fox jump over the lazy dog");

// insert the document
collection.insert(doc);

// find a document
collection.find(where("firstName").eq("John").and(where("lastName").eq("Doe"));

// update the document
collection.update(where("firstName").eq("John"), createDocument("lastName", "Wick"));

// remove the document
collection.remove(doc);

// insert an object in repository
Employee emp = new Employee();
emp.setEmpId(124589);
emp.setFirstName("John");
emp.setLastName("Doe");

repository.insert(emp);

Create Indices

// create document index
collection.createIndex(indexOptions(IndexType.NON_UNIQUE), "firstName", "lastName"); // compound index
collection.createIndex(indexOptions(IndexType.FULL_TEXT), "note"); // full-text index

// create object index. It can also be provided via annotation
repository.createIndex(indexOptions(IndexType.NON_UNIQUE), "firstName");

Query a Collection

DocumentCursor cursor = collection.find(
    where("firstName").eq("John")               // firstName == John
    .and(
        where("data").elemMatch("$".lt(4))      // AND elements of data array is less than 4
            .and(
                where("note").text("quick")     // AND note field contains string 'quick' using full-text index
        )       
    )
);

for (Document document : cursor) {
    // process the document
}

// get document by id
Document document = collection.getById(nitriteId);

// query an object repository and create the first result
Cursor<Employee> cursor = repository.find(where("firstName").eq("John"));
Employee employee = cursor.firstOrNull();

Transaction

try (Session session = db.createSession()) {
    try (Transaction transaction = session.beginTransaction()) {
        NitriteCollection txCol = transaction.getCollection("test");

        Document document = createDocument("firstName", "John");
        txCol.insert(document);

        transaction.commit();
    } catch (TransactionException e) {
        transaction.rollback();
    }
}

Schema Migration

Migration migration1 = new Migration(Constants.INITIAL_SCHEMA_VERSION, 2) {
    @Override
    public void migrate(InstructionSet instructions) {
        instructions.forDatabase()
            // make a non-secure db to secure db
            .addUser("test-user", "test-password");

        // create instructions for existing repository
        instructions.forRepository(OldClass.class, "demo1")

            // rename the repository (in case of entity name changes)
            .renameRepository("migrated", null)

            // change datatype of field empId from String to Long and convert the values
            .changeDataType("empId", (TypeConverter<String, Long>) Long::parseLong)

            // change id field from uuid to empId
            .changeIdField(Fields.withNames("uuid"), Fields.withNames("empId"))

            // delete uuid field
            .deleteField("uuid")
    
            // rename field from lastName to familyName
            .renameField("lastName", "familyName")

            // add new field fullName and add default value as - firstName + " " + lastName
            .addField("fullName", document -> document.get("firstName", String.class) + " "
                + document.get("familyName", String.class))

            // drop index on firstName
            .dropIndex("firstName")

            // drop index on embedded field literature.text
            .dropIndex("literature.text")

            // change data type of embedded field from float to integer and convert the values 
            .changeDataType("literature.ratings", (TypeConverter<Float, Integer>) Math::round);
    }
};

Migration migration2 = new Migration(2, 3) {
    @Override
    public void migrate(InstructionSet instructions) {
        instructions.forCollection("test")
            .addField("fullName", "Dummy Name");
    }
};

MVStoreModule storeModule = MVStoreModule.withConfig()
    .filePath("/temp/employee.db")
    .compressHigh(true)
    .build();

db = Nitrite.builder()
    .loadModule(storeModule)
    
    // schema versioning is must for migration
    .schemaVersion(2)

    // add defined migration paths
    .addMigrations(migration1, migration2)
    .openOrCreate();

Import/Export Data

// Export data to json file

// create export options
ExportOptions exportOptions = new ExportOptions();
// set the nitrite factory
exportOptions.setNitriteFactory(() -> openDb("test.db"));
// set the collections to export
exportOptions.setCollections(List.of("first"));
// set the repositories to export
exportOptions.setRepositories(List.of("org.dizitart.no2.support.data.Employee", "org.dizitart.no2.support.data.Company"));
// set the keyed repositories to export
exportOptions.setKeyedRepositories(Map.of("key", Set.of("org.dizitart.no2.support.data.Employee")));
// create an exporter with export options
Exporter exporter = Exporter.withOptions(exportOptions);
exporter.exportTo("test.json");

// Import data from the file

// create import options
ImportOptions importOptions = new ImportOptions();
// set the nitrite factory
importOptions.setNitriteFactory(() -> openDb("new-test.db"));
// create an importer with import options
Importer importer = Importer.withOptions(importOptions);
importer.importFrom("test.json");

More details are available in the guide.

Release Notes

Release notes are available here.

Documentation

Reference API

Document

JavaDoc

Build

To build and test Nitrite, ensure you have JDK 11 (or higher) and Maven 3 installed.

git clone https://github.com/nitrite/nitrite-java.git
cd nitrite-java
mvn clean install

Support / Feedback

For issues with, questions about, or feedback create a discussion.

Bugs / Feature Requests

Think you’ve found a bug? Want to see a new feature in the Nitrite? Please open an issue here. But before you file an issue please check if it is already existing or not.

Maintainers

  • Anindya Chatterjee

Contributors

This project exists thanks to all the people who contribute. For more details please visit CONTRIBUTING.md.

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. Become a sponsor for this project.

Presentation & Talks

Idan Sheinberg has given a talk on Nitrite at Kotlin Everywhere - TLV Edition meetup on October 27, 2019. Please find his presentation here.

Special Thanks

nitrite-java's People

Contributors

almondtools avatar anidotnet avatar dependabot-preview[bot] avatar dependabot[bot] avatar dunemaster avatar eddywm avatar mage1k99 avatar mend-bolt-for-github[bot] avatar monkeywithacupcake avatar morrica avatar mrnuggelz avatar niro avatar null-dev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nitrite-java's Issues

Error when using ObjectFilters.ALL with an empty ObjectRepository

Hi, i'm creating an empty database and after that trying to get the count of the elements for an objectRepository (at begin i expect a 0) returns this exception:

I'm using something like this:
int count = objectRepository.find(ObjectFilters.ALL).totalCount()

And this is a part of the exception:

 org.dizitart.no2.exceptions.ValidationException: NO2.1076: filter can not be null
                                                                                    at org.dizitart.no2.util.ValidationUtils.notNull(ValidationUtils.java:76)
                                                                                    at org.dizitart.no2.objects.DefaultObjectRepository.find(DefaultObjectRepository.java:188)

Create database using File

Currently database can be created only using filePath string. It will be good to allow creating db with File parameter

Is this db works fine with Kotlin and data classes?

Hello! Thanks for this project, it looks promising!

I'd like to use this library in production, is it reliable?
Can I use it with Kotlin's data classes?
Will it work properly for Android and iOS development with Intel MOE?

Objects with same @id duplication

I have such data class:

data class CaObject(
        @Id val localId: UUID,
        val name: String,
        val icon: CaoIcon,
        val lastInteractionDateTime: LocalDateTime?,
        ...
)

And I sometimes upsert values in ObjectsRepository like this:

    fun put(caObject: CaObject) = Completable.fromCallable {
        repository.update(caObject, true)
    }

I don't know in what excact moment or after what exact operation(maybe upsert isn't guilty), but now I have two objects with the same @Id in one collection:

[CaObject(localId=2deb953a-6218-4ba7-8309-96fca93bfa0d, name=1234, ...),
CaObject(localId=2deb953a-6218-4ba7-8309-96fca93bfa0d, name=1234 ...)]

Error saving object with a _id string field

Hi, currently i'm working with mongodb at the server side and nitrite for the client.

I'm trying to save a object from the server but the problem is that mongo manages the keys as strings like "592e4aa9ae0bb8df019a1155"

When i trying to save an object from a class that contains a string _id field, the insertion fails.

org.dizitart.no2.exceptions.InvalidIdException: NO2.6002: invalid _id found null
                                                                                    at org.dizitart.no2.Document.getId(Document.java:139)
                                                                                    at org.dizitart.no2.internals.DataService.insert(DataService.java:65)
                                                                                    at org.dizitart.no2.internals.NitriteService.insert(NitriteService.java:279)
                                                                                    at org.dizitart.no2.internals.DefaultNitriteCollection.insert(DefaultNitriteCollection.java:154)
                                                                                    at org.dizitart.no2.objects.DefaultObjectRepository.insert(DefaultObjectRepository.java:107)

Looks like a need to find a workaround to this, but maybe is a good idea to document this problem.

Problem within ObjectFilters.eq("field", "value")

when i try this
repository.find(ObjectFilters.eq("name", "jhonny")).size() // result = 0 but it should return 3
it works as expected when I use ObjectFilters.regex instead of ObjectFilters.eq
repository.find(ObjectFilters.regex("name", "jhonny")).size() // result = 3 -> GOOD

PS:he database contains three record with name = "jhonny"
and
repository.find(ObjectFilters.eq("name", "jhonny")).firstOrDefault() // always return null
all other operations works fine except equality filter!
Could you please help me.

Can't replace document's property with null

When I trying to upsert object to DB:

caObjectsDao.put(caObject.copy(
    output2Status = CaoStatus(false, currentDateTime),
    output2TriggerEndDateTime = null
))

output2Status changed, but output2TriggerEndDateTime still has the previous value instead of null.

ConcurrentModificationException

Hi again :)

I got an exception on updating object in repository:

Caused by: java.util.ConcurrentModificationException
    at java.util.HashMap$HashIterator.nextEntry(HashMap.java:851)
    at java.util.HashMap$KeyIterator.next(HashMap.java:885)
    at org.dizitart.no2.internals.ChangeEventBus.post(ChangeEventBus.java:40)
    at org.dizitart.no2.internals.ChangeEventBus.post(ChangeEventBus.java:30)
    at org.dizitart.no2.internals.DataService.notify(DataService.java:262)
    at org.dizitart.no2.internals.DataService.update(DataService.java:197)
    at org.dizitart.no2.internals.NitriteService.update(NitriteService.java:361)
    at org.dizitart.no2.internals.DefaultNitriteCollection.update(DefaultNitriteCollection.java:322)
    at org.dizitart.no2.objects.DefaultObjectRepository.update(DefaultObjectRepository.java:144)
    at org.dizitart.no2.objects.DefaultObjectRepository.update(DefaultObjectRepository.java:128)

As mentioned in documentation

ObjectRepository is thread-safe for concurrent use.

What can be the reason of such exception?

Is possible to exclude fields in POJOs for ObjectRepository?

Hi, currently i'm using the ObjectRepository to store my objects, but there is an annotation to exclude specific fields to be stored?
For instance this is my class:

class MyInformation {
       @Id
       String myId;
       String valueA;
       String valueB;
}

I want to exclude the valueB field to be stored and use it just in run time (because it changes dynamically at run time).

Database unexpectedly closed after successful upsert

Database unexpectedly closed after successful upsert:

  1. Try to get data from collection. Collection is empty so null is returned.
  2. Insert data into collection using upsert. Write Result show 1 record affected.
  3. Try to get data from collection. Database is suddenly closed
    Illegal State Exception: This store is closed
fun dbSave(key : String, data : Any)
{
    val doc = Document();
    doc.put(DB_COL_ID, key);
    doc.put(DB_COL_REC, data);
    var result = database.getCollection(DbTable.Singletons.tblName).update(eq(DB_COL_ID, key), doc, updateOptions(true));

}

fun <T> dbLoad(key : String) : T?
{
    return database.getCollection(DbTable.Singletons.tblName)
            .find(eq(DB_COL_ID, key))
            .map({x->
                if(x != null)
                {
                    x[DB_COL_REC] as T
                } else null})
            .firstOrNull();
}

Export all repositories

RepositoryRegistry contains only repositories previously opened using getRepository(). It means, that Export tool does not write all database entities into the exported json file.

Maybe this is a feature, not a bug, but the question is - how can I export all database objects including that not accessed until Export is running?

Error running datagate

Hi,

Im triying to run the datagate docker image but i have this error message

root@ubuntu-512mb-tor1-01:/home/server/repo/ci/datagate# docker run -p 9090:8080 --name container-datagate-test datagate-test
container_linux.go:247: starting container process caused "exec: "./datagate.sh": permission denied"
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: "./datagate.sh": permission denied".

About replication: Question

Hello @anidotnet

I am wondering if I have to the always the format "user-collection@userId"as the name of the replicated collection name? What is the point about that?

Thanks

Nitrite Failing on Second Write to DB

First write into a table is successful but second write is throwing an exception. I'm not closing the database explicitly:

val DB_PATH: String = "iotdb.database";
val database : Nitrite by lazy {makeOrOpenDb()};

private fun makeOrOpenDb(): Nitrite
{
    val db = Nitrite.builder().compressed().filePath(DB_PATH).openOrCreate();
    return db;
}

fun saveReport(report: IIotReport)
{
    database.getCollection(DbTable.Reports.tblName).insert(makeDocument(report));
}

private fun makeDocument(report: IIotReport): Document
{
    val doc = Document();
    doc.put(DB_COL_TYPE, report.type.toString());
    doc.put(DB_COL_DATE, report.date.time);
    doc.put(DB_COL_SN, report.serialNumber.toString());
    doc.put(DB_COL_REC, report);
    return doc;
}

Error:


Hello 2
Report Server Running
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Received RangerReport(date=Sun Oct 29 14:43:36 UTC 2017, serialNumber=5342313135371314340C, range=300, digitalIo=-1, atodOne=-1, atodTwo=-1)
Received RangerReport(date=Sun Oct 29 14:43:47 UTC 2017, serialNumber=5342313135371314340C, range=300, digitalIo=-1, atodOne=-1, atodTwo=-1)
Exception in thread "Report Server" java.lang.IllegalStateException: This store is closed [1.4.196/4]
        at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765)
        at org.h2.mvstore.MVStore.checkOpen(MVStore.java:2409)
        at org.h2.mvstore.MVStore.openMap(MVStore.java:443)
        at org.h2.mvstore.MVStore.openMap(MVStore.java:427)
        at org.dizitart.no2.store.NitriteMVStore.openMap(NitriteMVStore.java:87)
        at org.dizitart.no2.Nitrite.getCollection(Nitrite.java:116)
        at db.MainDbKt.saveReport(MainDb.kt:167)
        at iotsvr.ReportSvrKt$startReportServer$reportSvr$1.invoke(ReportSvr.kt:93)
        at iotsvr.ReportSvrKt$startReportServer$reportSvr$1.invoke(ReportSvr.kt)
        at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:18)
Caused by: java.lang.IllegalStateException: Last block not stored, possibly due to out-of-memory [1.4.196/3]
        at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765)
        at org.h2.mvstore.MVStore.storeNowTry(MVStore.java:1094)
        at org.h2.mvstore.MVStore.storeNow(MVStore.java:1050)
        at org.h2.mvstore.MVStore.commitAndSave(MVStore.java:1039)
        at org.h2.mvstore.MVStore.writeInBackground(MVStore.java:2489)
        at org.h2.mvstore.MVStore$BackgroundWriterThread.run(MVStore.java:2720)

java.nio issues with a lot of Android devices

Hello! Im the android developer for StatsRoyale and an happy user of nitrite db for multiple projects.
Due to the amount of installs that StatsRoyale have (almost 500k) you can imagine that as well, the amount of problems that pop to light are a lot.

I would like to report this problem coming from a lot of different devices:

Fatal Exception: java.lang.NoClassDefFoundError: Failed resolution of: Ljava/nio/file/Paths; at org.dizitart.no2.NitriteBuilder.openOrCreateInternal(NitriteBuilder.java:404) at org.dizitart.no2.NitriteBuilder.openOrCreate(NitriteBuilder.java:313) at com.overwolf.statsroyale.DBController$1.run(DBController.java:60) at android.os.Handler.handleCallback(Handler.java:743) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:150) at android.os.HandlerThread.run(HandlerThread.java:61) Caused by java.lang.ClassNotFoundException: Didn't find class "java.nio.file.Paths" on path: DexPathList[[zip file "/data/app/com.overwolf.statsroyale-1/base.apk"],nativeLibraryDirectories=[/data/app/com.overwolf.statsroyale-1/lib/arm, /vendor/lib, /system/lib]] at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56) at java.lang.ClassLoader.loadClass(ClassLoader.java:511) at java.lang.ClassLoader.loadClass(ClassLoader.java:469) at org.dizitart.no2.NitriteBuilder.openOrCreateInternal(NitriteBuilder.java:404) at org.dizitart.no2.NitriteBuilder.openOrCreate(NitriteBuilder.java:313) at com.overwolf.statsroyale.DBController$1.run(DBController.java:60) at android.os.Handler.handleCallback(Handler.java:743) at android.os.Handler.dispatchMessage(Handler.java:95) at android.os.Looper.loop(Looper.java:150) at android.os.HandlerThread.run(HandlerThread.java:61)

nio is not inside any android prior to 7 or even 8 because got introduced with JDK 7 (Android still uses 6 as far as i know) but not everyone is getting the issue.. I would guess only certain probably due to manufacturer customizations.

Can we go for standard File instead?

Id Annotation isn't inherited by child classes

Perhaps an annotation similar to @MappedSuper in JPA so if I am using the same id field in each entity I can just extend a parent class instead of having to mark up an id field in every class.

Otherwise great library, much better than CouchbaseLite. Especially since it doesn't require any native libraries.

Problems running datagate docker image 1.0.1

Hi,

I'm will simplify the problem, lets try to run the 1.0 docker image of datagate and enter to the command line:

image

But if you try this using the 1.0.1 docker image of datagate we have a problem!:

image

Datagate lock collection problem

Hi,
Im testing an android application with datagate, currently my server is exposing the datagate ports in the Dockerfile:

ENV DATAGATE_HTTP_PORT "8080"
ENV DATAGATE_HTTPS_PORT "8443"
ENV DATAGATE_MONITOR_PORT "9090"
ENV DATAGATE_KEY_STORE "keystore.jks"
ENV DATAGATE_KEY_PASSWORD "s3cret"

And this is an example of my test:

        NitriteCollection booksCollection =  db.getCollection("Books");

        if (booksCollection.find(eq("name", "Lost Symbol")).firstOrDefault() == null) {
            Document newBook = new Document();
            newBook.put("name", "Lost Symbol");
            newBook.put("author", "Dan Brown");
            booksCollection.insert(newBook);
        }

        DataGateClient dataGateClient = new DataGateClient("http://myserver:8080")
                .withAuth("admin", "somePassword");

        DataGateSyncTemplate syncTemplate = new DataGateSyncTemplate(dataGateClient, "Books@admin");

        // create sync handle
        SyncHandle syncHandle = Replicator.of(db)
                .forLocal(booksCollection)
                // a DataGate sync template implementation
                .withSyncTemplate(syncTemplate)
                // replication attempt delay of 1 sec
                .delay(timeSpan(10, TimeUnit.SECONDS))
                // both-way replication
                .ofType(ReplicationType.PUSH)
                // sync event listener
                .withListener(new SyncEventListener() {
                    @Override
                    public void onSyncEvent(SyncEventData eventInfo) {
                        Log.i("datagate", "Collection: " + eventInfo.getCollectionName() + " ErrorMessage: " + eventInfo.getError().getMessage() + " EventType: " + eventInfo.getEventType().toString());
                    }
                })
                .configure();

        // start sync in the background using handle
        syncHandle.startSync();

But after run the application i have this message logged:

I/datagate: Collection: Books ErrorMessage: NO2.11012: error while acquiring lock to remote collection EventType: REPLICATION_ERROR

I'm not sure if i'm using the right port (using 9090 give no response)
Also i will like to understand the relation between:

ENV DATAGATE_KEY_STORE "keystore.jks"
ENV DATAGATE_KEY_PASSWORD "s3cret"

I generated the keystore.jks field with the default configuration.

Thanks for your help!

Aggregation support

I just found about Nitrite and it's awesome. Any plans to support aggregations similar to what mongodb does?

Android and Java 8

Hi!

Google don't include Java 8 Time API in Android's standard library.
How can we deal with? I use threetenabp backport, will it work with nitrite?

java.lang.NoClassDefFoundError: Failed resolution of: Ljava/time/Instant;
                                             at com.fasterxml.jackson.datatype.jsr310.JavaTimeModule.<init>(JavaTimeModule.java:139)
                                             at org.dizitart.kno2.KNO2JacksonMapper.<init>(KNO2JacksonMapper.kt:35)
                                             at org.dizitart.kno2.Builder.createNitriteBuilder$potassium_nitrite(Builder.kt:114)
                                             at org.dizitart.kno2.BuilderKt.nitrite(Builder.kt:141)
                                             at org.dizitart.kno2.BuilderKt.nitrite$default(Builder.kt:138)

Jackson serialization error on Localdate

Hello,

I am facing serialization while using LocalDate in my POJO.
I realized I need to configure custom mapper. The problem is that I don't user spring web and the Jackson2ObjectMapperBuilder class can be found there.

 com.fasterxml.jackson.databind.JsonMappingException: No suitable constructor found for type [simple type, class java.time.LocalDate]: can not instantiate from JSON object (missing default constructor or creator, or perhaps need to add/enable type information?)
 at [Source: N/A; line: -1, column: -1] (through reference chain: com.hakilikoura.neerem.rdv.RendezVous["dateCreation"])

the text obove is the issue I am facing.

Could I have some help to solve my problem?

Thanks

mongodb connection

I read about Nitrite few days ago and I decided to try it...

I went into some troubles while trying to connect nitrite datagate with mongo db following the instruction on the website. I am not very comfortable with docker but I built the images from the docker file, run it. Before I launch and mongo using also docker. my mongoDB is running on localhost:27017 and I my datagate Dockerfile is :

FROM dizitart/nitrite-datagate

##COPY keystore.jks /

## Connection details (Replace with your own values)
ENV DATAGATE_HOST "0.0.0.0"
ENV DATAGATE_HTTP_PORT "8080"
ENV DATAGATE_HTTPS_PORT "8443"
ENV DATAGATE_MONITOR_PORT "9090"
ENV DATAGATE_KEY_STORE "keystore.jks"
ENV DATAGATE_KEY_PASSWORD "s3cret"

## Mongo connection details (Replace with your own values)
ENV DATAGATE_MONGO_HOST "127.0.0.1"
ENV DATAGATE_MONGO_PORT "27017"
ENV DATAGATE_MONGO_USER "admin"
ENV DATAGATE_MONGO_PASSWORD "password"
ENV DATAGATE_MONGO_DATABASE "demo"

## Starts the server
RUN ["chmod", "+x", "./datagate.sh"]
ENTRYPOINT [ "./datagate.sh" ]

I also did:

db.getCollection('no2.datagate.users').insert({
    userName: "admin",
    password: "password",
    accountNonExpired: true,
    accountNonLocked: true,
    credentialsNonExpired: true,
    enabled: true,
    authorities: ["ADMIN"]
});

I firstly encountered a problem with jks, that's why I commented it.

Then when I run the datagate, I got :

2017-06-08 02:21:32.800  INFO 6 --- [           main] org.mongodb.driver.cluster               : Cluster created with settings {hosts=[127.0.0.1:27017], mode=SINGLE, requir
edClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2017-06-08 02:21:32.870  INFO 6 --- [127.0.0.1:27017] org.mongodb.driver.cluster               : Exception in monitor thread while connecting to server 127.0.0.1:27017

com.mongodb.MongoSocketOpenException: Exception opening socket
        at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[mongo-java-driver-3.4.2.jar!/:na]
        at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[mongo-java-driver-3.4.2.jar!/:na]
        at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) ~[mongo-java-driver-3.4.2.jar!/:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_121]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_121]
        at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_121]
        at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[mongo-java-driver-3.4.2.jar!/:na]
        at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[mongo-java-driver-3.4.2.jar!/:na]
        ... 3 common frames omitted

2017-06-08 02:21:32.928  INFO 6 --- [           main] org.mongodb.driver.cluster               : No server chosen by WritableServerSelector from cluster description Cluster
Description{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.Mongo
SocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]}. Waiting for 30000 ms before timing out

Could you help me set that up to continue the exploration.

I really like the idea behind nitrite and I was looking for something like that.

Error with java.nio.file.Paths

Hi,
I tried to create a simple application using nitrite:

  • Updated gradle

  • Created sample project with a simple db creation code

And after run the aplication i got this:

05-30 01:45:39.096 27695-27702/? E/art: Failed sending reply to debugger: Broken pipe
05-30 01:45:39.193 27695-27695/? E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.example.newtype.nitritetest, PID: 27695
java.lang.NoClassDefFoundError: Failed resolution of: Ljava/nio/file/Paths;
at org.dizitart.no2.NitriteBuilder.openOrCreateInternal(NitriteBuilder.java:387)
at org.dizitart.no2.NitriteBuilder.openOrCreate(NitriteBuilder.java:338)
at com.example.newtype.nitritetest.MainActivity.startDatabase(MainActivity.java:26)
at com.example.newtype.nitritetest.MainActivity.onCreate(MainActivity.java:18)
at android.app.Activity.performCreate(Activity.java:6705)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1119)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2664)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2772)
at android.app.ActivityThread.-wrap12(ActivityThread.java)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1515)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:241)
at android.app.ActivityThread.main(ActivityThread.java:6217)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:865)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:755)
Caused by: java.lang.ClassNotFoundException: Didn't find class "java.nio.file.Paths" on path: DexPathList[[zip file "/data/app/com.example.newtype.nitritetest-2/base.apk"],nativeLibraryDirectories=[/data/app/com.example.newtype.nitritetest-2/lib/arm64, /system/lib64, /vendor/lib64]]
at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
at java.lang.ClassLoader.loadClass(ClassLoader.java:380)
at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
at org.dizitart.no2.NitriteBuilder.openOrCreateInternal(NitriteBuilder.java:387) 
at org.dizitart.no2.NitriteBuilder.openOrCreate(NitriteBuilder.java:338) 
at com.example.newtype.nitritetest.MainActivity.startDatabase(MainActivity.java:26) 
at com.example.newtype.nitritetest.MainActivity.onCreate(MainActivity.java:18) 
at android.app.Activity.performCreate(Activity.java:6705) 
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1119) 
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2664) 
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2772) 
at android.app.ActivityThread.-wrap12(ActivityThread.java) 
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1515) 
at android.os.Handler.dispatchMessage(Handler.java:102) 
at android.os.Looper.loop(Looper.java:241) 
at android.app.ActivityThread.main(ActivityThread.java:6217) 
at java.lang.reflect.Method.invoke(Native Method) 
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:865) 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:755) 

NO2.11011: error while submitting changes to remote collection

I am trying to use nitrite database with a project I am working on and it needs to be connected to the datagate server.

I set up the users and clients when syncing starts it throwing this error.
on the client side
org.dizitart.no2.exceptions.SyncException: NO2.11011: error while submitting changes to remote collection

on the server side
org.dizitart.no2.exceptions.InvalidOperationException: NO2.3009: document id is an auto generated value and can not be 771985705443479845

and I have a question. do I need to create the collection manually on mongo.

Thanks for help

Android specific documentation change

Add android specific initialisation code in the quick example.

Excerpt from gitter:

Janeo @eyronic Jun 22 01:43
You may want change the quick examples of initialize database to a working one.
Having .filePath("/tmp/test.db") wont work and it would confuse newbies because the error displayed is different
I got that error and it says im using java 1.6 when im using 1.8

multiuser webapplication with nitrite?

Hi Anindya,
my web prototype with primefaces + nitrite progress fine. Just one missing piece:
I intend to use nitrite as db for a small webapplication in my enterprise for internal use as well.
Max concurrent users would be 4-5 (non time critical), therefore I wrote a unit test with 5 threads that write entries into the db in parallel. Since nitrite does only allow one connection at time.
I tried it this way:

import bo.Region;
import static java.lang.Thread.sleep;
import java.util.UUID;
import org.dizitart.no2.Nitrite;
import org.dizitart.no2.NitriteCollection;
import org.dizitart.no2.objects.ObjectRepository;

public class TestThread extends Thread {

    String dbNameThread = "datamulti.db";
    Integer threadID;

    public TestThread(Integer threadID) {
        this.threadID = threadID;
    }

    public void run() {
        
        for (int i = 1; i <= 1000; i++) {
            try {
                System.out.println("Thread ["+ Integer.toString(threadID)+"] Schreibe " + Integer.toString(i));
                
                Region r = new Region();
                r.regionCode = Integer.toString(threadID) + "-" + Integer.toString(i);
                r.regionName = UUID.randomUUID().toString();
              
                Nitrite db = Nitrite.builder().compressed().filePath(dbNameThread).openOrCreate();
          
                ObjectRepository<Region> repRegion = db.getRepository(Region.class);
              
                repRegion.insert(r);

                db.close();
                db=null; >> close the db to free the connection for the other thread
                      
                sleep((int) (Math.random() * 100));
            } catch (Exception e) {
                System.out.println(e.getMessage());
            }
        }
    }
}

It doesn't work as expected. It is obvious since the threads block the db connection each other.

So I'm wondering what would be the best practice to allow several user to write data to db over one connection to the nitrite db, is there a mechanism similar to sqlite ?
Best regards.
Pascal Hoffmann

Geospatial Indexing

Would be nice if the documents/objects can support geospatial indexing/queries.

The property could be a geojson object representation and the querying/indexing can be performed using the JTS library.

Ex:

public class Place{ private String name; private GeoJson location; }

Cursor cursor = collection1.find().where("location").within(polygon); Cursor cursor = collection1.find().where("location").near(point,distance); Cursor cursor = collection1.find().where("location").intersects(geojson);

I've used JTS before, and there is a Jackson Library for serializing/deserializing the JTS Geometry objects to GeoJson. I think this feature would also provide universal value to others as it is becoming a pretty standard feature within DBs.

Thanks.

How to declare @Id in interface?

Interface:

@JsonTypeInfo(
        use = JsonTypeInfo.Id.NAME,
        include = JsonTypeInfo.As.PROPERTY,
        property = "type")
@JsonSubTypes(
        value = [
            JsonSubTypes.Type(CaoHistoryRecordNotification::class, name = "history-record-notification")
        ]
)
@Indices(value = arrayOf(
        Index(value = "caObjectLocalId", type = IndexType.NonUnique),
        Index(value = "checked", type = IndexType.NonUnique)
))
interface CaoNotification {
    @Id val localId: UUID
    val caObjectLocalId: UUID
    val checked: Boolean
    val dateTime: LocalDateTime
    val importance: CaoNotificationImportance
}

Data class:

data class CaoHistoryRecordNotification(
        override val localId: UUID,
        override val caObjectLocalId: UUID,
        override val checked: Boolean,
        override val dateTime: LocalDateTime,
        override val importance: CaoNotificationImportance,
        val localCaoHistoryRecordId: UUID
) : CaoNotification

But there is a problem, @Id annotation must be applied to fields only. How can I overcome it?
image

Without such annotation, Nitrite can't find id field for object when I trying to update repo

private val repository = nitrite.getRepository(CaoNotification::class.java)

Or repository of interfaces isn't supported?

StackOverflowError at org.h2.mvstore.MVMap.put

I am running into the following exception while doing a very basic batch of inserts:
java.lang.StackOverflowError
at org.h2.mvstore.MVMap.put(MVMap.java:185)
at org.h2.mvstore.MVMap.put(MVMap.java:211)
at org.h2.mvstore.MVMap.put(MVMap.java:209)
at org.h2.mvstore.MVMap.put(MVMap.java:211)
at org.h2.mvstore.MVMap.put(MVMap.java:209)
at org.h2.mvstore.MVMap.put(MVMap.java:211)
at org.h2.mvstore.MVMap.put(MVMap.java:209)
at org.h2.mvstore.MVMap.put(MVMap.java:211)
at org.h2.mvstore.MVMap.put(MVMap.java:209)
at org.h2.mvstore.MVMap.put(MVMap.java:211)
at org.h2.mvstore.MVMap.put(MVMap.java:209)
at org.h2.mvstore.MVMap.put(MVMap.java:211)

Steps to reproduce:

  • Download and unpack nitrite-stresstest.zip

  • Go the the unpack folder and run
    gradlew.bat test

  • Open build\test-results\test\TEST-org.dizitart.testing.NitriteStressTest.xml to see the result

The insert loop crashes after roughly 7200 records while I need to insert double of that. This i just a couple of MB's of data so this should not be a problem.

Joins

Support the ability to join records from other collections providing an output where the selected/join result is the document value.

Multi-language support for full-text search

Currently nitrite supports only English language for full-text search out-of-the-box. Add support for below languages in full-text search:

  • Afrikaans
  • Arabic
  • Armenian
  • Basque
  • Bengali
  • Breton
  • Bulgarian
  • Catalan
  • Chinese
  • Croatian
  • Czech
  • Danish
  • Dutch
  • English
  • Esperanto
  • Estonian
  • Finnish
  • French
  • Galician
  • German
  • Greek
  • Hausa
  • Hebrew
  • Hindi
  • Hungarian
  • Indonesian
  • Irish
  • Italian
  • Japanese
  • Korean
  • Kurdish
  • Latin
  • Latvian
  • Lithuanian
  • Malay
  • Marathi
  • Norwegian
  • Persian
  • Polish
  • Portuguese
  • Romanian
  • Russian
  • Sesotho
  • Slovak
  • Slovenian
  • Somali
  • Spanish
  • Swahili
  • Swedish
  • Tagalog
  • Thai
  • Turkish
  • Ukrainian
  • Urdu
  • Vietnamese
  • Yoruba
  • Zulu

Error accessing datagate portal

After starting the datagate docker image with the command:

docker run -p 8080:8080 -p 8443:8443 -p 9090:9090 --name container-test test

I cant access to the portal with any of the used ports:

image

image

image

ObjectRepository find method doesn't work when using Date type criteria

here is the sample code:

// Create an Object Repository
ObjectRepository<HtmlContentModel> repository = db.getRepository(HtmlContentModel.class);

// set time range
Calendar calendar=Calendar.getInstance();
calendar.set(year,1,1,0,0,0);
final Date startTime=calendar.getTime();
log.debug("startTime:{}",startTime);
calendar.set(year+1,1,1,0,0,0);
final Date endTime=calendar.getTime();
log.debug("endTime:{}",endTime);

// build object filter
ObjectFilter objectFilter=ObjectFilters.and(ObjectFilters.gte("publishTime",startTime),ObjectFilters.lt("publishTime",endTime));

// search and return object list by call cursor
org.dizitart.no2.objects.Cursor<HtmlContentModel> cursor=repository.find(objectFilter,FindOptions.sort("publishTime",SortOrder.Ascending));
cursor.toList(); //problem: it will return all of Objects
....

Joining multiple ALL Filters

Apparently this code

filter1 = Filters.ALL;
filter2 = Filters.ALL;
collection.find(Filters.and(filter1, filter2));

throws an invalid AND filter exception.
Wouldn't it be better if it was allowed, because in case multiple filters are used and each could be null/Filters.ALL, one has to check every single filter and combine the not-null ones with a lot of code.

Chunk ... no longer exists

I'm using Nitrite database to store a million documents and then perform different read operations on them (for benchmarking).
After storing the documents, while trying to read all documents via collection.find().iterator(), I get the following exception:

java.lang.IllegalStateException: Chunk 162 no longer exists [1.4.196/9]
	at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765)
	at org.h2.mvstore.MVStore.getChunkIfFound(MVStore.java:953)
	at org.h2.mvstore.MVStore.getChunk(MVStore.java:935)
	at org.h2.mvstore.MVStore.readPage(MVStore.java:1943)
	at org.h2.mvstore.MVMap.readPage(MVMap.java:741)
	at org.h2.mvstore.Page.getChildPage(Page.java:217)
	at org.h2.mvstore.Cursor.fetchNext(Cursor.java:150)
	at org.h2.mvstore.Cursor.next(Cursor.java:50)
	at org.dizitart.no2.internals.DocumentCursor$DocumentCursorIterator.nextMatch(DocumentCursor.java:115)
	at org.dizitart.no2.internals.DocumentIterator.next(DocumentIterator.java:54)
	at org.dizitart.no2.internals.DocumentIterator.next(DocumentIterator.java:32)
        ...

Then, the next smaller read operations return more values than they should.(Not related to this problem)
I create the database with the default options and three indexes, and it occurs with and without enabled compressing. The used version is 2.1.0.

Is a million documents too much for Nitrite? The same data and test cases work fine with pure MVStore. Inserting only ~340 entries works fine too.

Nitrite doesn't work with android 7.1.1

I want to use your library, but I have an exception on android 7.1.1

Version:

compile 'org.dizitart:nitrite:2.0.0'

Code:

      String path = Environment.getExternalStorageDirectory().getPath();
      File dbFile = new File(path + "/" + C.DB.DATABASE_FOLDER_NAME + "/" + C.DB.DATABASE_FILE_NAME);
      FileUtils.forceMkdirParent(dbFile);
      if (!dbFile.exists()) {
           dbFile.createNewFile();
      }
      db = Nitrite.builder()
                 .filePath(dbFile.getAbsolutePath())
                 .openOrCreate();`

Exception:

Caused by: org.dizitart.no2.exceptions.NitriteIOException: NO2.2001: database is already opened in other process
 at org.dizitart.no2.NitriteBuilder.openOrCreateInternal(NitriteBuilder.java:399)
 at org.dizitart.no2.NitriteBuilder.openOrCreate(NitriteBuilder.java:313)
 at ru.adhocapp.vocaberry.repository.UserDataRepository.<init>(UserDataRepository.java:39)
 at ru.adhocapp.vocaberry.repository.CourseWithUserData.withUserData(CourseWithUserData.java:25) 
 at ru.adhocapp.vocaberry.repository.FirebaseRepository$1.onDataChange(FirebaseRepository.java:56) 

I will be glad to any help.

ERROR NO2.8003 on Update object

Here is a code snippet using nitrite: first insert a region object,
then retrieve it from the database, in order to have an object with
autogenerated id, then i would like to update the object and store it again in the db.
With an non-autogenerated id it works fine, but not with a autogenerated.
I don't understand the error: //ERROR NO2.8003:
Since I retrieve the nitriteid from the db after having stored it in the db
my regSaved object should have an unique id and should be updateble
again. What wrong in this snippet ?

Nitrite db = Nitrite.builder().compressed().filePath(dbname).openOrCreate();

ObjectRepository<Region> repRegion = db.getRepository(Region.class);

Cursor<Region> cursorRegion1 = repRegion.find();
for (Region r : cursorRegion1) {
     repRegion.remove(r);
}
Region reg = new Region();
reg.regionName="BadenX";

WriteResult writeResult = repRegion.insert(reg); // insert a new region
NitriteId nitriteId = Iterables.firstOrDefault(writeResult); //gets the id of the new region after insert

Region regSaved = repRegion.getById(nitriteId); // retrieve the object
reg.regionName="Baden"; //update the name
repRegion.update(reg); //ERROR NO2.8003: update operation failed as no id value found for the object 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.