ververica/frocksdb

frocksdb's Issues

aarch64 support

Rocksdb 6.4.6 release now contains librocksdbjni-linux-aarch64.so for aarch64 platform. It's good for frocksdb to support it as well.

It's very easy to build the aarch64 package on an ARM machine:

export DEBUG_LEVEL=0 && make rocksdbjava

OpenLab community can donate ARM machine for frocksdb team to help build and release ARM package to maven center. Please let me know if you need one.

[Bug] Could not find option: compression_type

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

I'm trying to flip between a bulk-load mode and a regular mode. e.g. Disable auto_compaction and compression when the app starts up, load the data, then enable auto_compaction and compression.

Actual behavior

Invoking method: RocksDB.setOptions(...) got an exception.

Steps to reproduce the behavior

Version：

com.ververica
frocksdbjni
6.20.3-ververica-1.0

Error log:

org.rocksdb.RocksDBException: Could not find option: : compression_type

	at org.rocksdb.RocksDB.setOptions(Native Method)
	at org.rocksdb.RocksDB.setOptions(RocksDB.java:3577)

Demo:

        String dbPath = temp.newFolder().getPath();
        final DBOptions dbOptions = new DBOptions().setCreateIfMissing(true).setCreateMissingColumnFamilies(true)
            .setWalDir(dbPath).setStatsDumpPeriodSec(300).setStatistics(new Statistics());
        String columnFamilyName = "test_option_dynamic";
        ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
        columnFamilyOptions.setDisableAutoCompactions(true);
        columnFamilyOptions.setCompressionType(CompressionType.NO_COMPRESSION);

        ColumnFamilyDescriptor descriptor = new ColumnFamilyDescriptor(columnFamilyName.getBytes(), columnFamilyOptions);
        ColumnFamilyDescriptor defaultDes = new ColumnFamilyDescriptor("default".getBytes(), new ColumnFamilyOptions());

        final List<ColumnFamilyDescriptor> descriptors = new ArrayList<>();
        descriptors.add(descriptor);
        descriptors.add(defaultDes);
        final List<ColumnFamilyHandle> managedHandles = new ArrayList<>();

        RocksDB rocksDB = RocksDB.open(dbOptions, dbPath, descriptors, managedHandles);

        Assert.assertEquals(descriptors.size(), managedHandles.size());
        ColumnFamilyHandle columnFamilyHandle = managedHandles.get(0);

        rocksDB.put(columnFamilyHandle, "key1".getBytes(), "value1".getBytes());

        rocksDB.setOptions(
            columnFamilyHandle,
            MutableColumnFamilyOptions.builder()
                .setCompressionType(CompressionType.SNAPPY_COMPRESSION)
                .setDisableAutoCompactions(false)
                .build());

frocksdb 6.10 usage

I need to use the FIFO compation, and there is a configuration: compactionOptionsFIFO.setAllowCompaction, which is only available in
https://mvnrepository.com/artifact/io.github.myasuka/frocksdbjni 6.10 + version
I noticed there is a performance issue, facebook/rocksdb#5774
If I am not so concerned about performance, can I use it in my own job (flink 1.11)

Adding this commit to rocksdb version 7.9.2 reported an error when compiling with docker

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

Actual behavior

Steps to reproduce the behavior

Loading source files for package org.rocksdb...
Loading source files for package org.rocksdb.util...
Constructing Javadoc information...
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:34: error: package org.rocksdb.FlinkCompactionFilter does not exist
import org.rocksdb.FlinkCompactionFilter.StateType;
                                        ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:35: error: package org.rocksdb.FlinkCompactionFilter does not exist
import org.rocksdb.FlinkCompactionFilter.TimeProvider;
                                        ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:348: error: cannot find symbol
  private static class TestTimeProvider implements TimeProvider {
                                                   ^
  symbol:   class TimeProvider
  location: class FlinkCompactionFilterTest
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:133: error: package FlinkCompactionFilter does not exist
    final FlinkCompactionFilter.FlinkCompactionFilterFactory filterFactory;
                               ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:137: error: cannot find symbol
    private StateContext(StateType type, TimeProvider timeProvider, int timestampOffset) {
                                         ^
  symbol:   class TimeProvider
  location: class StateContext
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:159: error: package FlinkCompactionFilter does not exist
    FlinkCompactionFilter.Config createConfig(StateType type, int timestampOffset) {
                         ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:165: error: package FlinkCompactionFilter does not exist
        FlinkCompactionFilter.FlinkCompactionFilterFactory filterFactory) {
                             ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:231: error: cannot find symbol
    private FixedElementListStateContext(TimeProvider timeProvider) {
                                         ^
  symbol:   class TimeProvider
  location: class FixedElementListStateContext
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:236: error: package FlinkCompactionFilter does not exist
    FlinkCompactionFilter.Config createConfig(StateType type, int timestampOffset) {
                         ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:305: error: package FlinkCompactionFilter does not exist
    private static FlinkCompactionFilter.ListElementFilterFactory ELEM_FILTER_FACTORY =
                                        ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:308: error: cannot find symbol
    private NonFixedElementListStateContext(TimeProvider timeProvider) {
                                            ^
  symbol:   class TimeProvider
  location: class NonFixedElementListStateContext
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:313: error: package FlinkCompactionFilter does not exist
    FlinkCompactionFilter.Config createConfig(StateType type, int timestampOffset) {
                         ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:321: error: package FlinkCompactionFilter does not exist
        implements FlinkCompactionFilter.ListElementFilterFactory {
                                        ^
src/test/java/org/rocksdb/FlinkCompactionFilterTest.java:323: error: package FlinkCompactionFilter does not exist
      public FlinkCompactionFilter.ListElementFilter createListElementFilter() {
                                  ^
Standard Doclet version 1.8.0_392
Building tree for all the packages and classes...
Generating target/apidocs/org/rocksdb/AbstractCompactionFilter.html...
Generating target/apidocs/org/rocksdb/AbstractCompactionFilter.Context.html...
Generating target/apidocs/org/rocksdb/AbstractCompactionFilterFactory.html...
Generating target/apidocs/org/rocksdb/AbstractComparator.html...
Generating target/apidocs/org/rocksdb/AbstractEventListener.html...
Generating target/apidocs/org/rocksdb/AbstractEventListener.EnabledEventCallback.html...
src/test/java/org/rocksdb/AbstractTransactionTest.java:218: warning: [deprecation] multiGet(ReadOptions,List<ColumnFamilyHandle>,byte[][]) in Transaction has been deprecated
      assertThat(txn.multiGet(readOptions, cfList, keys)).isEqualTo(new byte[][] { null, null });
                    ^
src/test/java/org/rocksdb/AbstractTransactionTest.java:222: warning: [deprecation] multiGet(ReadOptions,List<ColumnFamilyHandle>,byte[][]) in Transaction has been deprecated
      assertThat(txn.multiGet(readOptions, cfList, keys)).isEqualTo(values);
                    ^
Generating target/apidocs/org/rocksdb/AbstractImmutableNativeReference.html...
src/test/java/org/rocksdb/AbstractTransactionTest.java:256: warning: [deprecation] multiGet(ReadOptions,byte[][]) in Transaction has been deprecated
      assertThat(txn.multiGet(readOptions, keys)).isEqualTo(new byte[][] { null, null });
                    ^
src/test/java/org/rocksdb/AbstractTransactionTest.java:260: warning: [deprecation] multiGet(ReadOptions,byte[][]) in Transaction has been deprecated
      assertThat(txn.multiGet(readOptions, keys)).isEqualTo(values);
                    ^
src/test/java/org/rocksdb/AbstractTransactionTest.java:322: warning: [deprecation] multiGetForUpdate(ReadOptions,List<ColumnFamilyHandle>,byte[][]) in Transaction has been deprecated
      assertThat(txn.multiGetForUpdate(readOptions, cfList, keys))
                    ^
Generating target/apidocs/org/rocksdb/AbstractMutableOptions.html...
src/test/java/org/rocksdb/AbstractTransactionTest.java:327: warning: [deprecation] multiGetForUpdate(ReadOptions,List<ColumnFamilyHandle>,byte[][]) in Transaction has been deprecated
      assertThat(txn.multiGetForUpdate(readOptions, cfList, keys))
                    ^
src/test/java/org/rocksdb/AbstractTransactionTest.java:344: warning: [deprecation] multiGetForUpdate(ReadOptions,byte[][]) in Transaction has been deprecated
      assertThat(txn.multiGetForUpdate(readOptions, keys)).isEqualTo(new byte[][]{null, null});
                    ^
src/test/java/org/rocksdb/AbstractTransactionTest.java:348: warning: [deprecation] multiGetForUpdate(ReadOptions,byte[][]) in Transaction has been deprecated
      assertThat(txn.multiGetForUpdate(readOptions, keys)).isEqualTo(values);
                    ^
Generating target/apidocs/org/rocksdb/AbstractMutableOptions.AbstractMutableOptionsBuilder.html...
Generating target/apidocs/org/rocksdb/AbstractNativeReference.html...
Generating target/apidocs/org/rocksdb/AbstractRocksIterator.html...
Generating target/apidocs/org/rocksdb/AbstractSlice.html...
Generating target/apidocs/org/rocksdb/AbstractTableFilter.html...
src/test/java/org/rocksdb/BackupEngineOptionsTest.java:26: warning: [deprecation] none() in ExpectedException has been deprecated
  public ExpectedException exception = ExpectedException.none();

is frocksdb support store on hdfs?

I want store rocksdb data on hdfs, but i found rocksdb don't support on hdfs(facebook/rocksdb#9067). So i want konw is frocksdb support store on hdfs? Because flink rocksdbStateBackend is store data on hdfs.

java.lang.UnsatisfiedLinkError on mac m1

Caused by: java.lang.UnsatisfiedLinkError: /private/var/folders/9w/kbk7nnxd60jf5lqgp4vgm5kh0000gn/T/junit12882793858522803468/junit18283359092312814966/rocksdb-lib-c06728cfa5b110c51f7e8f16aa883bc0/librocksdbjni-osx.jnilib: dlopen(/private/var/folders/9w/kbk7nnxd60jf5lqgp4vgm5kh0000gn/T/junit12882793858522803468/junit18283359092312814966/rocksdb-lib-c06728cfa5b110c51f7e8f16aa883bc0/librocksdbjni-osx.jnilib, 0x0001): tried: '/private/var/folders/9w/kbk7nnxd60jf5lqgp4vgm5kh0000gn/T/junit12882793858522803468/junit18283359092312814966/rocksdb-lib-c06728cfa5b110c51f7e8f16aa883bc0/librocksdbjni-osx.jnilib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e'))
at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2442)
at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2498)
at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694)
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627)
at java.base/java.lang.Runtime.load0(Runtime.java:768)
at java.base/java.lang.System.load(System.java:1837)
at org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:79)
at org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:57)
at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:856)
... 18 more

releated issue in rocksdb facebook/rocksdb#9721
releated issue in flink https://issues.apache.org/jira/browse/FLINK-26876

How to compile a cross-platform jar locally

https://github.com/ververica/frocksdb/blob/FRocksDB-6.20.3/java/RELEASE.md
I compiled frocksdb on centos7.6 according to the document, but encountered many errors.

First, I ran the make jclean clean rocksdbjavastaticreleasedocker command and encountered the following error
CFLAGS='-fPIC ' LDFLAGS=' ' ./configure --static --archs='-arch x86_64' && make
Checking for gcc...
Compiler error reporting is too harsh for ./configure (perhaps remove -Werror).
** ./configure aborting.

and then delete all -arch x86_64, and encountered the error
docker run --rm --name rocksdb_linux_x86-be --platform linux/386 --attach stdin --attach stdout --attach stderr --volume /root/.m2:/root/.m2:ro --volume pwd:/rocksdb-host:ro --volu-java-target --env DEBUG_LEVEL=0 evolvedbinary/rocksjava:centos6_x86-be /rocksdb-host/java/crossbuild/docker-build-linux-centos.sh
unknown flag: --platform

Is there any detailed document or CI for reference? thank you

doesn't work on window 10 21H2 version

Caused by: java.lang.Exception: Exception while creating StreamOperatorStateContext.
at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state backend for CoBroadcastWithKeyedOperator_ac9a901f2ba35c2ad13a5f3044240476_(2/12) from any of the 1 provided restore options.
at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
... 11 more
Caused by: java.io.IOException: Could not load the native RocksDB library
at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882)
at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
at org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:93)
at org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
at org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
... 13 more
Caused by: java.lang.UnsatisfiedLinkError: D:\checkpoint\rocksdb-lib-317ecc348d5fbe7c64fa10384c7eb8fc\librocksdbjni-win64.dll: librocksdbjni-win64.dll: A dynamic link library (DLL) initialization routine failed

not work with vc 2015/vc 2017 of win10 H21H2，

but it's working on win10 H20H2

flink v1.14.4