pmem/llpl

Low Level Persistence Library

llpl's Issues

JVM crashed when creating PersistentHeap

Stack: [0x00007f9810c8d000,0x00007f9810d8d000], sp=0x00007f9810d89ff8, free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libpthread.so.0+0x9d00] pthread_mutex_lock+0x0

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 839 com.intel.pmem.llpl.AnyHeap.nativeGetRoot(J)J (0 bytes) @ 0x00007f97f9625ba1 [0x00007f97f9625b60+0x41]
J 835 C1 com.intel.pmem.llpl.AnyHeap.(Ljava/lang/String;J)V (98 bytes) @ 0x00007f97f9626b14 [0x00007f97f96264c0+0x654]
J 833 C1 com.intel.pmem.llpl.PersistentHeap.createHeap(Ljava/lang/String;J)Lcom/intel/pmem/llpl/PersistentHeap; (317 bytes) @ 0x00007f97f962f9ac [0x00007f97f962eaa0+0xf0c]
v ~StubRoutines::call_stub

simple makes then will see UT failed. what is root cause?

root@ubuntu:/home/wayne/llpl# make && make tests
g++ -O3 -DNDEBUG -fPIC -I/usr/lib/jvm/java-1.8.0-openjdk-amd64/include -I/usr/lib/jvm/java-1.8.0-openjdk-amd64/include/linux -o target/cppbuild/lib_llpl_Transaction.o -c src/main/cpp/lib_llpl_Transaction.cpp
g++ -O3 -DNDEBUG -fPIC -I/usr/lib/jvm/java-1.8.0-openjdk-amd64/include -I/usr/lib/jvm/java-1.8.0-openjdk-amd64/include/linux -o target/cppbuild/persistent_heap.o -c src/main/cpp/persistent_heap.cpp
g++ -O3 -DNDEBUG -fPIC -I/usr/lib/jvm/java-1.8.0-openjdk-amd64/include -I/usr/lib/jvm/java-1.8.0-openjdk-amd64/include/linux -o target/cppbuild/lib_llpl_AnyMemoryBlock.o -c src/main/cpp/lib_llpl_AnyMemoryBlock.cpp
g++ -O3 -DNDEBUG -fPIC -I/usr/lib/jvm/java-1.8.0-openjdk-amd64/include -I/usr/lib/jvm/java-1.8.0-openjdk-amd64/include/linux -o target/cppbuild/lib_llpl_AnyHeap.o -c src/main/cpp/lib_llpl_AnyHeap.cpp
g++ -Wl,-soname,target/cppbuild/libllpl.so -o target/cppbuild/libllpl.so target/cppbuild/lib_llpl_Transaction.o target/cppbuild/persistent_heap.o target/cppbuild/lib_llpl_AnyMemoryBlock.o target/cppbuild/lib_llpl_AnyHeap.o -fPIC -O3 -DNDEBUG -shared -lpmem -lpmemobj -Wl,-rpath,/usr/local/lib:/usr/local/lib64
mkdir -p target/classes
mkdir -p target/test_classes
/usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/javac -Xlint:unchecked -proc:none -XDenableSunApiLintControl -d target/classes -cp target/classes:lib src/main/java/lib/llpl/package-info.java src/main/java/lib/llpl/TransactionalHeap.java src/main/java/lib/llpl/TransactionException.java src/main/java/lib/llpl/MemoryBlock.java src/main/java/lib/llpl/Range.java src/main/java/lib/llpl/UnboundedMemoryBlock.java src/main/java/lib/llpl/PersistentUnboundedMemoryBlock.java src/main/java/lib/llpl/Heap.java src/main/java/lib/llpl/AbstractMemoryBlock.java src/main/java/lib/llpl/AbstractPersistentMemoryBlock.java src/main/java/lib/llpl/TransactionalMemoryBlock.java src/main/java/lib/llpl/TransactionalUnboundedMemoryBlock.java src/main/java/lib/llpl/Stats.java src/main/java/lib/llpl/AbstractTransactionalMemoryBlock.java src/main/java/lib/llpl/Transaction.java src/main/java/lib/llpl/AnyMemoryBlock.java src/main/java/lib/llpl/PersistentMemoryBlock.java src/main/java/lib/llpl/HeapException.java src/main/java/lib/llpl/AnyHeap.java src/main/java/lib/llpl/PersistentHeap.java
Note: src/main/java/lib/llpl/AnyHeap.java uses internal proprietary API that may be removed in a future release.
Note: Recompile with -Xlint:sunapi for details.
/usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/javac -Xlint:unchecked -proc:none -XDenableSunApiLintControl -cp target/classes:lib:src/examples/array src/examples/array/.java; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/javac -Xlint:unchecked -proc:none -XDenableSunApiLintControl -cp target/classes:lib:src/examples/string_store src/examples/string_store/.java;
/usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/javac -Xlint:unchecked -proc:none -XDenableSunApiLintControl -cp target/classes:lib:src -d target/test_classes src/test/java/lib/llpl/*.java;
/usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/CopyMemoryTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/PersistentMemoryBlockTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/MemoryBlockCollectionTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/MemoryBlockFreeTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/MemoryBlockEqualityTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/MemoryBlockTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/MultipleHeapTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/MultipleTransactionalHeapTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/SetMemoryTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/TransactionTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/TransactionalMemoryBlockTest; /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -ea -cp target/classes:lib:target/test_classes -Djava.library.path=target/cppbuild lib/llpl/UnboundedMemoryBlockTest;
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.Heap.(Heap.java:16)
at lib.llpl.Heap.getHeap(Heap.java:30)
at lib.llpl.CopyMemoryTest.(CopyMemoryTest.java:14)
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.PersistentHeap.(PersistentHeap.java:20)
at lib.llpl.PersistentHeap.getHeap(PersistentHeap.java:34)
at lib.llpl.PersistentMemoryBlockTest.main(PersistentMemoryBlockTest.java:12)
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.Heap.(Heap.java:16)
at lib.llpl.Heap.getHeap(Heap.java:30)
at lib.llpl.MemoryBlockCollectionTest.main(MemoryBlockCollectionTest.java:15)
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.Heap.(Heap.java:16)
at lib.llpl.Heap.getHeap(Heap.java:30)
at lib.llpl.MemoryBlockFreeTest.main(MemoryBlockFreeTest.java:12)
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.Heap.(Heap.java:16)
at lib.llpl.Heap.getHeap(Heap.java:30)
at lib.llpl.MemoryBlockEqualityTest.main(MemoryBlockEqualityTest.java:18)
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.Heap.(Heap.java:16)
at lib.llpl.Heap.getHeap(Heap.java:30)
at lib.llpl.MemoryBlockTest.main(MemoryBlockTest.java:13)
================================= All MultipleHeap tests passed ================================
================================= All MultipleTransactionalHeap tests passed ===================
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.Heap.(Heap.java:16)
at lib.llpl.Heap.getHeap(Heap.java:30)
at lib.llpl.SetMemoryTest.main(SetMemoryTest.java:14)
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.Heap.(Heap.java:16)
at lib.llpl.Heap.getHeap(Heap.java:30)
at lib.llpl.TransactionTest.main(TransactionTest.java:12)
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.TransactionalHeap.(TransactionalHeap.java:20)
at lib.llpl.TransactionalHeap.getHeap(TransactionalHeap.java:34)
at lib.llpl.TransactionalMemoryBlockTest.main(TransactionalMemoryBlockTest.java:13)
Exception in thread "main" java.lang.RuntimeException: Failed to open heap.
at lib.llpl.AnyHeap.(AnyHeap.java:53)
at lib.llpl.Heap.(Heap.java:16)
at lib.llpl.Heap.getHeap(Heap.java:30)
at lib.llpl.UnboundedMemoryBlockTest.main(UnboundedMemoryBlockTest.java:12)
Makefile:86: recipe for target 'tests' failed
make: *** [tests] Error 1
root@ubuntu:/home/wayne/llpl#

com.intel.pmem.llpl.HeapException: Failed to allocate memory of size

com.intel.pmem.llpl.HeapException: Failed to allocate memory of size 2097152
        at com.intel.pmem.llpl.MemoryAccessor.lambda$new$0(MemoryAccessor.java:65)
        at com.intel.pmem.llpl.MemoryAccessor.<init>(MemoryAccessor.java:77)
        at com.intel.pmem.llpl.AnyMemoryBlock.<init>(AnyMemoryBlock.java:34)
        at com.intel.pmem.llpl.AbstractMemoryBlock.<init>(AbstractMemoryBlock.java:16)
        at com.intel.pmem.llpl.MemoryBlock.<init>(MemoryBlock.java:25)
        at com.intel.pmem.llpl.Heap.allocateMemoryBlock(Heap.java:226)
        at org.apache.hadoop.hbase.regionserver.PersistentMemstoreLAB.allocateBytes(PersistentMemstoreLAB.java:102)
        at org.apache.hadoop.hbase.regionserver.TestPersistentMemstoreLAB.testLABLargeAllocation(TestPersistentMemstoreLAB.java:74)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)

I got this exception when I tested HBase, using LLPL.
The Heap size was 8MB on my server.

Use growing heap: Heap.createHeap(heapPath, heapSize);

Any hints about where went wrong?

Do we have a plan to release the 1.2.0 to maven repository?

Do we have a plan to release the 1.2.0 to maven repository?

Does pmem support pagecache?

The doc of llpl points out that MemoryBlock maybe not persist if not flush manually, is there any api support pagecache so that the OS will handle this. Thank you.

[Proposal] Add support for DirectByteBuffer

Background:
For I/O intensive java application, we usually use DirectByteBuffer to manage data flow from(to) network and file, Java.nio support using DirectBuffer and channels to avoid copy data into java heap. DirectByteBuffer is a common used byte buffer. When using pmem to accerate I/O with java application, we should avoid copying data into java heap, which will cause GC and latency. Therefore, llpl lib should support write from DirectByteBuffer to PMEM.

Details:
DirectByteBuffer will hold a address(located out of Java heap), we could pass this address to PMDK copy_from_array function, and change DirectByteBuffer 's position after copy success.

The NullPointerException error when I use LLPL

Hello everyone，
When building and running tests in the LLPL, I found the error below:

java.lang.ExceptionInInitializerError
	at com.intel.pmem.llpl.IntArrayTests.initialize(IntArrayTests.java:23)
Caused by: java.lang.NullPointerException
	at com.intel.pmem.llpl.IntArrayTests.initialize(IntArrayTests.java:23)

I locate the error in the com.intel.pmem.llpl.Util<loadLibrary()>
because the libname it got is /com/intel/pmem/llpl/linux/i386/libllpl.so
but there is only one libllpl.so in the /com/intel/pmem/llpl/linux/amd64
And the version of my machine is SMP Mon Dec 23 10:15:49 CST 2019 x86_64 x86_64 x86_64 GNU/Linux
so, the os.arch may be wrong.

And when I used this library in existing java applications with maven, I got the same error.
Thanks.

Compile error when making native lib

Issue

Missing declarations on libpmempool.h to build native lib.

Environment Information

• OS version: Ubuntu 18.04 LTS
• Kernel version: 4.15.0-147
• libpmempool version: 1.4.1

Details

chriptus13@chriptus13:~/llpl/target/cppbuild$ make
[ 16%] Building CXX object CMakeFiles/llpl.dir/com_intel_pmem_llpl_AnyHeap.cpp.o
/home/chriptus13/llpl/src/main/cpp/com_intel_pmem_llpl_AnyHeap.cpp: In function ‘jint Java_com_intel_pmem_llpl_AnyHeap_nativeHeapExists(JNIEnv*, jobject, jstring)’:
/home/chriptus13/llpl/src/main/cpp/com_intel_pmem_llpl_AnyHeap.cpp:164:10: error: use of enum ‘pmempool_feature’ without previous declaration
     enum pmempool_feature feature = PMEMPOOL_FEAT_CKSUM_2K;
          ^~~~~~~~~~~~~~~~
/home/chriptus13/llpl/src/main/cpp/com_intel_pmem_llpl_AnyHeap.cpp:164:37: error: ‘PMEMPOOL_FEAT_CKSUM_2K’ was not declared in this scope
     enum pmempool_feature feature = PMEMPOOL_FEAT_CKSUM_2K;
                                     ^~~~~~~~~~~~~~~~~~~~~~
/home/chriptus13/llpl/src/main/cpp/com_intel_pmem_llpl_AnyHeap.cpp:164:37: note: suggested alternative: ‘PMEMPOOL_POOL_TYPE_BLK’
     enum pmempool_feature feature = PMEMPOOL_FEAT_CKSUM_2K;
                                     ^~~~~~~~~~~~~~~~~~~~~~
                                     PMEMPOOL_POOL_TYPE_BLK
/home/chriptus13/llpl/src/main/cpp/com_intel_pmem_llpl_AnyHeap.cpp:166:15: error: ‘pmempool_feature_query’ was not declared in this scope
     int ret = pmempool_feature_query(native_string, feature, 0);
               ^~~~~~~~~~~~~~~~~~~~~~
/home/chriptus13/llpl/src/main/cpp/com_intel_pmem_llpl_AnyHeap.cpp:166:15: note: suggested alternative: ‘pmempool_transform’
     int ret = pmempool_feature_query(native_string, feature, 0);
               ^~~~~~~~~~~~~~~~~~~~~~
               pmempool_transform
CMakeFiles/llpl.dir/build.make:62: recipe for target 'CMakeFiles/llpl.dir/com_intel_pmem_llpl_AnyHeap.cpp.o' failed
make[2]: *** [CMakeFiles/llpl.dir/com_intel_pmem_llpl_AnyHeap.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/llpl.dir/all' failed
make[1]: *** [CMakeFiles/llpl.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Failed to allocate memory of size xxx

import com.intel.pmem.llpl.Heap;
import com.intel.pmem.llpl.MemoryBlock;


Heap heap = Heap.createHeap(heapName, 8388608);
MemoryBlock block = heap.allocateMemoryBlock(5242880, false);

It will cause a problem, the error log:

Exception in thread "main" com.intel.pmem.llpl.HeapException: Failed to allocate memory of size 5242880
	at com.intel.pmem.llpl.MemoryAccessor.lambda$new$0(MemoryAccessor.java:63)
	at com.intel.pmem.llpl.MemoryAccessor.<init>(MemoryAccessor.java:75)
	at com.intel.pmem.llpl.AnyMemoryBlock.<init>(AnyMemoryBlock.java:34)
	at com.intel.pmem.llpl.AbstractMemoryBlock.<init>(AbstractMemoryBlock.java:16)
	at com.intel.pmem.llpl.MemoryBlock.<init>(MemoryBlock.java:25)
	at com.intel.pmem.llpl.Heap.allocateMemoryBlock(Heap.java:221)

What are the causes of this error?