kevoree-modeling/mwdb

Many-World Database repository

mwdb's Issues

Store the polynom in a property - Feature request

Currently, the polynomial value of a PolynomialNode is accessed by parsing the polynomial attribute of the JSON generated by the toString method
eg. {"world":0,"time":1468504965,"id":10"polynomial":"29.373022419094397+(-0.019802106705884497*t)"}

It would be useful to retrieve directly this value through a method or a property

eg.thePolynomialNodeInstance.get("polynomial") or thePolynomialNodeInstance.getPolynomial()

[JS] Récupérer les décimales des attributs des nodes

Je crée un nouveau noeud dans le graphe :

...
var sensor0 = graph.newNode(0, 0);
    sensor0.set("name", "sensor0");
    sensor0.set("value", 30.0);
...

Puis quand je récupère l'attribut en JS j'obtiens : "value: 30" au lieu de 30.0

Tree-graph layout + Directed edge relationship

• Many ML algorithms are structured in trees and subtrees, a tree/hierarchical layout of the graph would be better and much more useful to debug.
• Directed edges relationships are interesting to show as well

Windows failed test

Failed tests: testReadFilesOnFile(org.mwg.ImporterTest): expected:<[/D:/workspace/mwDB/plugins/importer/target/test-classes/smarthome/]readme.md> but was:<[D:\workspace\mwDB\plugins\importer\target\test-classes\smarthome]readme.md>

KDTree OffHeap Sigsev

When we use KDTree on offheap memory, a SIGSEV signal is trhown:

# JMH 1.13 (released 9 days ago)
# VM version: JDK 1.8.0_66, VM 25.66-b17
# VM invoker: /opt/jdk/jdk1.8.0_66/jre/bin/java
# VM options: <none>
# Warmup: <none>
# Measurement: 1 iterations, single-shot each, 100000 calls per op
# Timeout: 5 min per iteration
# Threads: 1 thread
# Benchmark mode: Single shot invocation time
# Benchmark: org.kevoree.mwg.benchmark.ml.KDTree.benchKDTree
# Parameters: (cacheSize = 10000000, useHeap = false)

Forking using command: [/opt/jdk/jdk1.8.0_66/jre/bin/java, -XX:CompileCommandFile=/tmp/jmh2579372353722079712compilecommand, -cp, bench/target/mwg-benchmark.jar, org.openjdk.jmh.runner.ForkedMain, 127.0.0.1, 34736]
# Run progress: 0.00% complete, ETA 00:00:00
# Fork: 1 of 1
Iteration   1: #
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fe9cf62c68c, pid=11443, tid=140642134656768
#
# JRE version: Java(TM) SE Runtime Environment (8.0_66-b17) (build 1.8.0_66-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x93468c]  ParMarkBitMap::mark_obj(HeapWord*, unsigned long)+0xcc
#
# Core dump written. Default location: /var/jenkins_home/workspace/KDTree_OffHeap_Sigsev/benchmark/core or core.11443
#
# An error report file with more information is saved as:
# /var/jenkins_home/workspace/KDTree_OffHeap_Sigsev/benchmark/hs_err_pid11443.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
<forked VM failed with exit code 134>
<stdout last='20 lines'>
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fe9cf62c68c, pid=11443, tid=140642134656768
#
# JRE version: Java(TM) SE Runtime Environment (8.0_66-b17) (build 1.8.0_66-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x93468c]  ParMarkBitMap::mark_obj(HeapWord*, unsigned long)+0xcc
#
# Core dump written. Default location: /var/jenkins_home/workspace/KDTree_OffHeap_Sigsev/benchmark/core or core.11443
#
# An error report file with more information is saved as:
# /var/jenkins_home/workspace/KDTree_OffHeap_Sigsev/benchmark/hs_err_pid11443.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
</stdout>
<stderr last='20 lines'>
</stderr>

The error occurs on Debian 8 machine (docker), not on OSX computer.

There is a build process on build server to reproduce this bug.

Small memory leak on Polynomial read

There is a small memory leak with KDTree :)
Bench result:

# JMH 1.13 (released 9 days ago)
# VM version: JDK 1.8.0_66, VM 25.66-b17
# VM invoker: /opt/jdk/jdk1.8.0_66/jre/bin/java
# VM options: <none>
# Warmup: <none>
# Measurement: 1 iterations, single-shot each, 1000000 calls per op
# Timeout: 5 min per iteration
# Threads: 1 thread
# Benchmark mode: Single shot invocation time
# Benchmark: org.kevoree.mwg.benchmark.ml.PolynomialRead.benchPolynomial
# Parameters: (cacheSize = 5000000, useHeap = true)

Forking using command: [/opt/jdk/jdk1.8.0_66/jre/bin/java, -XX:CompileCommandFile=/tmp/jmh98844555278615798compilecommand, -cp, bench/target/mwg-benchmark.jar, org.openjdk.jmh.runner.ForkedMain, 127.0.0.1, 41184]
# Run progress: 0.00% complete, ETA 00:00:00
# Fork: 1 of 100
Iteration   1: Aug 01, 2016 8:39:48 AM com.github.fommil.netlib.BLAS <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
Aug 01, 2016 8:39:48 AM com.github.fommil.netlib.BLAS <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
Aug 01, 2016 8:39:48 AM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
Aug 01, 2016 8:39:48 AM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
<failure>

java.lang.RuntimeException: Memory leak detected: startAvailableSpace=4999996; endAvailableSpace=4999994; diff= 2

It appears in heap and off heap memory, but only on build server (Debian 8.5/docker).

I created a small Jenkins build to reproduce it.

Performance bottleneck on relation add

After benchmarking, a big bottleneck currently is this code:

long[] incArray = new long[previous.length + 1];
System.arraycopy(previous, 0, incArray, 0, previous.length);
incArray[previous.length] = relatedNode.id();

In the add method of the AbstractNode class

where the increment of long[] is only by one.
We should increment by 30% each time.
the perf increases from 5,000 values/sec to 25,000,000 v/s

OffHeapTimeTreeChunk Load error

java.lang.RuntimeException: get: bad address 1*8 in null
	at org.mwg.memory.offheap.primary.OffHeapLongArray.get(OffHeapLongArray.java:77)
	at org.mwg.memory.offheap.OffHeapTimeTreeChunk.internal_insert(OffHeapTimeTreeChunk.java:776)
	at org.mwg.memory.offheap.OffHeapTimeTreeChunk.internal_load(OffHeapTimeTreeChunk.java:283)
	at org.mwg.memory.offheap.OffHeapTimeTreeChunk.load(OffHeapTimeTreeChunk.java:228)
	at org.mwg.WSServer$7.on(WSServer.java:272)
	at org.mwg.WSServer$7.on(WSServer.java:263)
	at org.mwg.memory.offheap.OffHeapChunkSpace$1.on(OffHeapChunkSpace.java:170)
	at org.mwg.memory.offheap.OffHeapChunkSpace$1.on(OffHeapChunkSpace.java:160)
	at org.mwg.LevelDBStorage.get(LevelDBStorage.java:63)
	at org.mwg.memory.offheap.OffHeapChunkSpace.getOrLoadAndMark(OffHeapChunkSpace.java:160)
	at org.mwg.WSServer.process_put(WSServer.java:263)

with the following buffer

#rQO4I#A;////////+;A;A#C|Y|A|BA:Y2l0aWVz:CiJIlv:bmFtZQ:ZvUW:aW5kZXg:L9lGk:ZGlzdGFuY2U:iYxfq:Ym91bmRfbWlu:CJHb/f:Ym91bmRfbWF4:CJHcG7:cG9zaXRpb25z:DLgRjU:bnVtYmVy:B7Tklv:cG9zaXRpb24:BZJTJS:YmlrZV9zdGFuZHM:DbSr0L:U0xPVFNfTlVNQkVS:DWDUab:UEVSSU9EX1NJWkU:CRuC9+:c3RhdGlvbl9wcm9maWxl:CBQITF:YXZhaWxhYmxlX2Jpa2Vz:BX0pd4:YXZhaWxhYmxlX2Jpa2Vfc3RhbmRz:BjLLX3:c3RhdHVz:BqZGAd:c3RhdGlvbnM:CdKEt+:Y29udHJhY3RfbmFtZQ:9GgTR:YWRkcmVzcw:CI0MgZ:YmFua2luZw:oLc+1:Ym9udXM:LMkA+:bGF0:0M+:bG5n:0ZK:ZWdyYXBo:Cbi4/v:c3RyYXRlZ3k:DVH0bm:X2ZlYXR1cmVzTmI:BvxlDg:X3RvdGFs:CumYN3:X21pbg:Wasm:X21heA:WalK:X3N1bQ:WdsY:X3N1bVNxdWFyZQ:BT/kxv:dmFsdWU:NWSLi#G;////////9;////////+;C#U#A;A;////////9;EAAAAAC#G|E|ZvUW|QW1pZW5z|a|DLgRjU|C:EAAAAAG|c|CdKEt+|C:Xp4046vu1:EAAAAAI#C;A;////////+;EAAAAAC#C:////////9#C;A;////////9;EAAAAAC#C:////////9#E;A;A;EAAAAAC#C:A:////////9#A;A;////////9;////////8#C|U|L9lGk|C:CiJIlv:EAAAAAE#C;A;////////+;////////8#C:////////9#C;A;////////9;////////8#C:////////9#E;A;A;////////8#C:A:////////9#A;A;////////9;EAAAAAE#C|c|L9lGk|C:xOmUkPQp8:EAAAAAC#C;A;////////+;EAAAAAE#C:////////9#C;A;////////9;EAAAAAE#C:////////9#E;A;A;EAAAAAE#DcBchh|C:A:////////9#A;A;////////9;EAAAAAG#K|I|iYxfq|C|M|CJHb/f|E:wFGg:wGGg|M|CJHcG7|E:QFGg:QGGg|i|Cbi4/v|C$Q%G%A%C%G%E%A%G%C%A%M%G%E:wFGg:wGGg%M%I%E:QFGg:QGGg%e%K%m:QAA:QCA:P/A:QEI8vW9ZxFd:QACXI9DaL42:f/I:f/I:f/I:f/I:f/I:f/I:f/I:f/I:f/I:f/I:f/I:f/I:f/I:f/I%g%M%W:C:Q:C:EAAAAAI:D:D:D:D:D:D:D%G%A%C|I|DVH0bm|A#C;A;////////+;EAAAAAG#C:////////9#C;A;////////9;EAAAAAG#C:////////9#E;A;A;EAAAAAG#Dt/7MZ|C:A:////////9#A;A;////////9;EAAAAAI#O|E|B7Tklv|MDAwMw|a|BZJTJS|C:EAAAAAK|a|DbSr0L|C:EAAAAAM|a|CBQITF|C:EAAAAAO|a|BX0pd4|C:EAAAAAQ|a|BjLLX3|C:EAAAAAS|a|BqZGAd|C:EAAAAAU#C;A;////////+;EAAAAAI#C:////////9#C;A;////////9;EAAAAAI#E:////////9:q76Juyg#E;A;A;EAAAAAI#C:A:////////9#A;A;////////9;EAAAAAK#A#C;A;////////+;EAAAAAK#C:////////9#C;A;////////9;EAAAAAK#E:////////9:q76Juyg#E;A;A;EAAAAAK#C:A:////////9#A;A;////////9;EAAAAAM#A#C;A;////////+;EAAAAAM#C:////////9#C;A;////////9;EAAAAAM#E:////////9:q76Juyg#E;A;A;EAAAAAM#C:A:////////9#A;A;////////9;EAAAAAO#E|I|DWDUab|FQ|G|CRuC9+|BIGQgA#C;A;////////+;EAAAAAO#BIGQgA|C:////////9#C;A;////////9;EAAAAAO#E:////////9:q7mezgA#E;A;A;EAAAAAO#DyNara|C:A:////////9#A;A;////////9;EAAAAAQ#A#C;A;////////+;EAAAAAQ#C:////////9#C;A;////////9;EAAAAAQ#W:////////9:q76Juyg:q7/CVoA:q7/dx+Q:q8APAXQ:q8BNF/w:q8BNdbw:q8BSEWw:q8CrNbw:q8DTxIw:q8D3lng#E;A;A;EAAAAAQ#C:A:////////9#A;A;////////9;EAAAAAS#A#C;A;////////+;EAAAAAS#C:////////9#C;A;////////9;EAAAAAS#W:////////9:q76Juyg:q7/CVoA:q7/dx+Q:q8APAXQ:q8BNF/w:q8BNdbw:q8BSEWw:q8Cq3dg:q8DTxIw:q8D3lng#E;A;A;EAAAAAS#C:A:////////9#A;A;////////9;EAAAAAU#A#C;A;////////+;EAAAAAU#C:////////9#C;A;////////9;EAAAAAU#E:////////9:q76Juyg#E;A;A;EAAAAAU#C:A:////////9#A;A;q76Juyg;EAAAAAK#E|K|0M+|QEI8vW9ZxFd|K|0ZK|QACXI9DaL42#A;A;q76Juyg;EAAAAAQ#C|I|NWSLi|G#A;A;q76Juyg;EAAAAAS#C|I|NWSLi|i#A;A;q76Juyg;EAAAAAU#C|E|NWSLi|T1BFTg#A;A;q76Juyg;EAAAAAM#C|I|NWSLi|o

Time & Index issues

I'm indexing a collection of stations.
The first station is at time 3 and the index is dynamically creted with this object.
Then I have a station at time 1 and the index triggers a runtime exception

java.lang.RuntimeException: Index creation failed, cache is probably full !!!
    at org.mwg.core.CoreGraph$4.on(CoreGraph.java:403)
    at org.mwg.core.CoreGraph$4.on(CoreGraph.java:399)
    at org.mwg.core.MWGResolver$1$1.on(MWGResolver.java:130)
    at org.mwg.core.MWGResolver$1$1.on(MWGResolver.java:119)
    at org.mwg.core.chunk.heap.HeapChunkSpace.getOrLoadAndMark(HeapChunkSpace.java:120)
    at org.mwg.core.MWGResolver$1.on(MWGResolver.java:119)
    at org.mwg.core.MWGResolver$1.on(MWGResolver.java:112)
    at org.mwg.core.chunk.heap.HeapChunkSpace.getOrLoadAndMark(HeapChunkSpace.java:120)
    at org.mwg.core.MWGResolver.lookup(MWGResolver.java:112)
    at org.mwg.core.CoreGraph$10.on(CoreGraph.java:615)
    at org.mwg.core.CoreGraph$10.on(CoreGraph.java:587)
    at org.mwg.core.MWGResolver$1$1$1$1.on(MWGResolver.java:188)
    at org.mwg.core.MWGResolver$1$1$1$1.on(MWGResolver.java:149)
    at org.mwg.core.chunk.heap.HeapChunkSpace.getOrLoadAndMark(HeapChunkSpace.java:120)
    at org.mwg.core.MWGResolver$1$1$1.on(MWGResolver.java:149)
    at org.mwg.core.MWGResolver$1$1$1.on(MWGResolver.java:133)
    at org.mwg.core.chunk.heap.HeapChunkSpace.getOrLoadAndMark(HeapChunkSpace.java:120)
    at org.mwg.core.MWGResolver$1$1.on(MWGResolver.java:133)
    at org.mwg.core.MWGResolver$1$1.on(MWGResolver.java:119)
    at org.mwg.core.chunk.heap.HeapChunkSpace.getOrLoadAndMark(HeapChunkSpace.java:120)
    at org.mwg.core.MWGResolver$1.on(MWGResolver.java:119)
    at org.mwg.core.MWGResolver$1.on(MWGResolver.java:112)
    at org.mwg.core.chunk.heap.HeapChunkSpace.getOrLoadAndMark(HeapChunkSpace.java:120)
    at org.mwg.core.MWGResolver.lookup(MWGResolver.java:112)
    at org.mwg.core.CoreGraph.getIndexOrCreate(CoreGraph.java:587)
    at org.mwg.core.CoreGraph.index(CoreGraph.java:399)
    at org.mwg.core.task.ActionIndexOrUnindexNode.eval(ActionIndexOrUnindexNode.java:45)
    at org.mwg.core.task.CoreTaskContext.continueTask(CoreTaskContext.java:347)
    at org.mwg.core.task.ActionPrint.eval(ActionPrint.java:20)
    at org.mwg.core.task.CoreTaskContext.continueTask(CoreTaskContext.java:347)
    at org.mwg.core.task.CoreTaskContext.continueWith(CoreTaskContext.java:294)
    at org.mwg.plugin.geojson.ActionNewNodeFromJson.eval(ActionNewNodeFromJson.java:34)
    at org.mwg.core.task.ActionPlugin.eval(ActionPlugin.java:82)
    at org.mwg.core.task.CoreTaskContext.continueTask(CoreTaskContext.java:347)
    at org.mwg.core.task.CoreTaskContext.continueWith(CoreTaskContext.java:294)
    at org.mwg.core.task.ActionFromVar.eval(ActionFromVar.java:30)
    at org.mwg.core.task.CoreTaskContext.execute(CoreTaskContext.java:361)
    at org.mwg.core.task.CoreTask$3.run(CoreTask.java:454)
    at org.mwg.core.scheduler.TrampolineScheduler.dispatch(TrampolineScheduler.java:31)
    at org.mwg.core.task.CoreTask.executeWith(CoreTask.java:400)
    at org.mwg.core.task.CoreTask.execute(CoreTask.java:374)
    at org.mwg.GeoJsonTest.update(GeoJsonTest.java:168)
    at org.mwg.GeoJsonTest.lambda$baseTest$0(GeoJsonTest.java:68)
    at org.mwg.GeoJsonTest$$Lambda$1/935044096.on(Unknown Source)
    at org.mwg.core.CoreGraph$2$1$1.on(CoreGraph.java:305)
    at org.mwg.core.CoreGraph$2$1$1.on(CoreGraph.java:254)
    at org.mwg.core.BlackHoleStorage.get(BlackHoleStorage.java:31)
    at org.mwg.core.CoreGraph$2$1.on(CoreGraph.java:254)
    at org.mwg.core.CoreGraph$2$1.on(CoreGraph.java:236)
    at org.mwg.core.BlackHoleStorage.lock(BlackHoleStorage.java:58)
    at org.mwg.core.CoreGraph$2.on(CoreGraph.java:236)
    at org.mwg.core.CoreGraph$2.on(CoreGraph.java:233)
    at org.mwg.core.BlackHoleStorage.connect(BlackHoleStorage.java:50)
    at org.mwg.core.CoreGraph.connect(CoreGraph.java:233)
    at org.mwg.GeoJsonTest.baseTest(GeoJsonTest.java:41)
    at org.mwg.GeoJsonTest.main(GeoJsonTest.java:208)

[Scala] 'then' might be used as an identifier in future version of scala

As shown here:

Adding toMap and putAll methods in Map(mwdb) - Feature Request

Hello,
It would be nice to have a toMap()and a putAll(List<Map.Entry<K,V>>) methods or equivalent in the classes extending Map such as LongLongMap

[REST Plugin] JSON does not accept NaN values

A NaN value in mwDB leads to a bad-formatted JSON:

{"world":0,"time":1464782931206,"id":274877907106,"name":"TEMP_444V","value":NaN,"type":"TEMPERATURE_INDOOR"}

KGraph.all() callback not called after unindex

I wrote a simpler example of what happens when a KGraph.all() method is called after an unindex call.

I you try the following test case, the "callback all 2" is not displayed.

But by removing the call to KGraph.unindex() everything works as expected.

package fr.mleduc.mwdb.test.mwdb;

import org.junit.Test;
import org.mwdb.*;

/**
 * Created by mleduc on 21/03/16.
 */
public class CallbackFail {

    @Test
    public void test0() {
        final KGraph graph = GraphBuilder.builder().buildGraph();
        graph.connect(new KCallback<Boolean>() {
            @Override
            public void on(Boolean result) {
                final KNode node0 = graph.newNode(0, 0);
                node0.attSet("x", KType.INT, 0);
                System.out.println("before index xs node0");
                graph.index("xs", node0, new String[]{"x"}, new KCallback<Boolean>() {
                    @Override
                    public void on(Boolean result) {
                        System.out.println("callback index xs node0");
                        System.out.println("before save");
                        graph.save(new KCallback<Boolean>() {
                            @Override
                            public void on(Boolean result) {
                                System.out.println("callback save");
                                System.out.println("before all 1");
                                graph.all(0, 1, "xs", new KCallback<KNode[]>() {
                                    @Override
                                    public void on(KNode[] result) {
                                        System.out.println("callback all 1");
                                        System.out.println("before unindex");
                                        graph.unindex("xs", result[0], new String[]{"x"}, new KCallback<Boolean>() {
                                            @Override
                                            public void on(Boolean result) {
                                                System.out.println("callback unindex");
                                                final KNode node1 = graph.newNode(0,1);
                                                node1.attSet("x", KType.INT, 1);
                                                System.out.println("before index xs node1");
                                                graph.index("xs", node1, new String[]{"x"}, new KCallback<Boolean>() {
                                                    @Override
                                                    public void on(Boolean result) {
                                                        System.out.println("callback index xs node1");
                                                        System.out.println("before all 2");
                                                        graph.all(0,2, "xs",  result1 -> {
                                                            System.out.println("callback all 2");
                                                            System.out.println("OK " + result1);
                                                        });
                                                    }
                                                });
                                            }
                                        });
                                    }
                                });
                            }
                        });
                    }
                });
            }
        });
    }
}

Task issue: Cannot refer directly to an array element

It is not possible to refer to an array element inside a task, directly.

E.g.
It's not possible to do
.loop("0","{{data[ i ]]}}"
but we have to define another variable as follows:

.then(taskContext -> {
  taskContext.defineVariable("g",
  taskContext.variable("data").get(
  (Integer) taskContext.variable("i").get(0)
 )
);
taskContext.continueTask();
})
.loop("0", "{{g}}")

Question: Commercial friendly license

This is a very interesting project. Do you consider switching to a commercial-friendly license to broaden its versatility ?