scalableminds/fossildb

Versioned Key-Value Store with RocksDB backend and gRPC API

fossildb's Issues

use new standard health check in python client

After implementing the standard grpc health check (#19) we should use it everywhere.

Multi-Get and Multi-Put with list of keys

Sometimes users know a specific list of keys they want to get or put. Currently, they have to send many grpcs, one per key (except for the special case of prefix scan gets). To reduce overhead, it would be nice to add API endpoints that directly support this.

message OptionalLong {
    optional uint64 value = 1;
}

message GetMultipleKeysByListRequest {
    required string collection = 1;
    repeated string keys = 2;
    repeated OptionalLong versions = 3;
}

// length of versions should either be equal to length of keys, in which case each individual version is considered optional, or 0 (empty list), in which case None is assumed for every key.

message GetMultipleKeysByListReply {
    required bool success = 1;
    repeated string errorMessage = 2;
    repeated bytes values = 4;
    repeated uint64 actualVersions = 5;
    repeated bool missingKeys = 6;
}

// if only some keys exist, values and actualVersions are be densified. missingValueIndices lists the keys for wich nothing has been found

(Note that the name GetMultipleKeys was unfortunately already taken by the prefix-scan-based request)

message PutMultipleKeysByListRequest {
    required string collection = 1;
    repeated string keys = 2;
    repeated OptionalLong versions = 3;
}

message PutMultipleKeysByListReply {
    required bool success = 1;
    optional string errorMessage = 2;
}

Any comments on the API draft? @normanrz @philippotto

Read Column Families from Options File, drop param "-c"

Double-check that health check is not blocked by other connections

We saw autoheal trigger during large requests. This may have been due to the health check responding slowly while that other request was being handled. We should double-check that these are as independent as possible.
I assume that it is already fine, as errors should have been far more common otherwise, but I’d rather check again.

Test backup and restore (in CircleCI)

add Smoketest

Test the docker image with some basic settings (maybe with client?)
See #8

Add healthcheck to dockerfile

The server dockerfile doesn't have a healthcheck yet. Using the healthcheck from the client is not possible because it is in a different image. Possible options that I see:

• Make a big docker image that has both client and server in one
• Hack a health check with grpcurl

Configure database through RocksDB options file

Can we initialize the fossildb through the RocksDB options file?
The benefit would be that we can customize all RocksDB options, including compaction parameters.
To create new databases, we could ship an options file with the project.

See https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/OptionsUtil.java#L89

once all usages are cleaned, remove old health check from protocol

Use standard grpc health check

To be consistent with the usual grpc health check tools (e.g. for kubernetes) we should use the default health library.
There is a Java implementation
and example.