pinterest/rocksplicator

RocksDB Replication

Rocksplicator

Important: Rocksplicator is an archived project that is no longer actively maintained or supported by Pinterest.

Rocksplicator is a set of C++ libraries and tools for building large scale RocksDB based stateful services. Its goal is to help application developers solve common difficulties of building large scale stateful services, such as data replication, request routing and cluster management. With Rocksplicator, application developers just need to focus on their application logics, and won't need to deal with data replication, request routing nor cluster management.

Rocksplicator includes:

1. RocksDB replicator (a library for RocksDB real-time data replication. It supports 3 different replication modes, i.e., async replication, semi-sync replication, and sync replication.)
2. Helix powered automated cluster management and recovery
3. Async fbthrift client pool and fbthrift request router
4. A stats library for maintaining & reporting server stats
5. A set of other small tool classes for building C++ services.

Online introduction videos

Introduction of Rocksplicator can be found in in our presentation at 2016 Annual RocksDB meetup at FB HQ and @Scale presentation (starting from 17:30).

Use cases

Currently, we have 9 different online services based on rocksplicator running at Pinterest, which consist of nearly 30 clusters, over 4000 hosts and process tens of PB data per day.

Prerequisities

The third-party dependencies of Rocksplicator can be found in docker/Dockerfile.

Get Started

Install docker

Docker is used for building Rocksplicator. Follow the Docker installation instructions to get Docker running on your system.

Build docker image

You can build your own docker image (if you want to change the docker file and test it locally).

cd docker && docker build -t rocksplicator-build .

Or pull the one we uploaded.

docker pull gopalrajpurohit/rocksplicator-build:librdkafka_1_4_0

Initialize submodules

cd rocksplicator && git submodule update --init

Build the libraries & tools

Get into the docker build environment. We are assuming the rocksplicator repo is under $HOME/code/, and $HOME/docker-root is an existing directory.

docker run -v <SOURCE-DIR>:/rocksplicator -v $HOME/docker-root:/root -ti gopalrajpurohit/rocksplicator-build:librdkafka_1_4_0 bash

Run the following command in the docker bash to build Rocksplicator:

cd /rocksplicator && mkdir -p build && cd build && cmake .. && make -j

Run Tests

Run the following command in the docker bash:

cd /rocksplicator && mkdir -p build && cd build && cmake .. && make -j && make test

How to build your own service based on RocksDB replicator & cluster management libraries.

There is an example counter service under examples/counter_service/, which demonstrated a typical usage pattern for RocksDB replicator.

Automated cluster management and recovery

Please check cluster_management directory for Helix powered automated cluster management and recovery.

Commands for cluster management (The following is for script driven cluster management, which has been deprecated)

The cluster mangement tool rocksdb_admin.py is under rocksdb_admin/tool/.

Before using the tool, we need to generate python client code for Admin interface as follows.

cd /rocksplicator/rocksdb_admin/tool/ && ./sync.sh

Create config file for a newly launched cluster.

host_file is a text file containing all hosts in the cluster. Each line is for a host in format "ip:port:zone". For example "192.168.0.101:9090:us-east-1c"

python rocksdb_admin.py new_cluster_name config --host_file=./host_file --segment=test --shard_num=1000 --overwrite

Ping all hosts in a cluster

python rocksdb_admin.py cluster_name ping

Remove a host from a cluster

python rocksdb_admin.py cluster_name remove_host "ip:port:zone"

Promote Master for shards currenty having Slaves only

python rocksdb_admin.py cluster_name promote

Add a host to a cluster

python rocksdb_admin.py cluster_name add_host "ip:port:zone"

Rebalance a cluster (Evenly distribute Masters)

python rocksdb_admin.py cluster_name rebalance

Load SST files from S3 to the cluster

python rocksdb_admin.py "cluster" load_sst "segment" "s3_bucket" "s3_prefix" --concurrency 64 --rate_limit_mb 64

Typical cluster management workflows (This has been deprecated, please check the new Helix powered solution in the cluster_management directory)

replacing a dead host

python rocksdb_admin.py cluster_name remove_host old_ip:old_port:zone_a
python rocksdb_admin.py cluster_name promote
python rocksdb_admin.py cluster_name add_host new_ip:new_port:zone_a
python rocksdb_admin.py cluster_name rebalance

License

Apache License, Version 2.0

rocksplicator's Issues

rocksplicator docker image build failed

I try to compiler rocksplicator docker image, but follower error is throw

[+] Building 3.6s (8/33)                                                                                                                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                      0.0s
 => => transferring dockerfile: 37B                                                                                                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/ubuntu:14.04                                                                                                                                                                                           0.5s
 => [ 1/30] FROM docker.io/library/ubuntu:14.04@sha256:63fce984528cec8714c365919882f8fb64c8a3edf23fdfa0b218a2756125456f                                                                                                                                   0.0s
 => CACHED [ 2/30] RUN apt-get -q -y update &&     apt-get -q -y install       python-software-properties       wget       git       software-properties-common                                                                                           0.0s
 => CACHED [ 3/30] RUN add-apt-repository -y ppa:ubuntu-toolchain-r/test                                                                                                                                                                                  0.0s
 => CACHED [ 4/30] RUN add-apt-repository ppa:openjdk-r/ppa && apt-get update                                                                                                                                                                             0.0s
 => ERROR [ 5/30] RUN wget 'https://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/cloudera.list'     -O /etc/apt/sources.list.d/cloudera.list &&     wget https://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key     -O archive.key &  3.0s

look like is https://archive.cloudera.com/cdh5/ubuntu/trusty/amd64/cdh/archive.key didn't execution success.

Please help, Thanks.

Kazoo - Cannot fetch index base URL https://pypi.python.org/simple/

Hi,

While running the following command I am getting missing dependency error:

python rocksdb_admin.py cluster_1 config --host_file=./host_file --segment=test --shard_num=1000 --overwrite

I got python-gevent dependency missing error:

**Traceback (most recent call last):

File "rocksdb_admin.py", line 28, in <module>

from kazoo import client

ImportError: No module named python-gevent**

I installed gevent: sudo apt install python-gevent

Then I got python kazoo dependency missing error:

  **Traceback (most recent call last):

  File "rocksdb_admin.py", line 28, in <module>

  from kazoo import client

  ImportError: No module named kazoo**

I am not able to install Kazoo:

pip install kazoo gives following error:

            **Downloading/unpacking kazoo

            Cannot fetch index base URL https://pypi.python.org/simple/

           Could not find any downloads that satisfy the requirement kazoo

           Cleaning up...

           No distributions at all found for kazoo

           Storing debug log for failure in /root/.pip/pip.log**

After that I tried installing Kazoo with pip3

pip3 install kazoo

pip3 worked

Now I tried running with python3:
python3 rocksdb_admin.py cluster_1 config --host_file=./host_file --segment=test --shard_num=1000 --overwrite

But this gives ImportError: No module named 'gevent' as python3 gevent is missing.
And now I am not able to get python3 gevent.

How shall I proceed? Any response is appreciated.

can rocksplicator work if disableWAL=true for the primary RocksDB?

Can rocksplicator work if disableWAL=true for the primary RocksDB?
in some case,we dsiable WAL for the performance reason，as I try the rocksdb API getUpdateSince，it won't got any update.Any suggestions？

stateMap could be null when starting a resource from the very beginning

https://github.com/pinterest/rocksplicator/blob/master/cluster_management/src/main/java/com/pinterest/rocksplicator/MasterSlaveStateModelFactory.java#L287

At the case, java raises.

Should it be?

if (stateMap != null) { for (Map.Entry<String, String> instanceNameAndRole : stateMap.entrySet()) {

Or did I miss how to create a resource from scratch?

Also the stateMap here: https://github.com/pinterest/rocksplicator/blob/master/cluster_management/src/main/java/com/pinterest/rocksplicator/MasterSlaveStateModelFactory.java#L153

questions

hello,i read have some questions after read the rocksplicator:
1)if the master is down,new master maybe promoted,perhaps the data maybe loss a little.
2)in the time of master promotion,the cluster maybe can't provide write service .
3)for one cluster，how it found the unavailable replication and recover it?ping perhaps only found node error.

Is it fine to initialize client_pool_ with an integer?

Is it fine to initialize client_pool_ with an integer?

https://github.com/pinterest/rocksplicator/blob/master/rocksdb_replicator/rocksdb_replicator.cpp#L41

https://github.com/pinterest/rocksplicator/blob/master/rocksdb_replicator/rocksdb_replicator.h#L217

How to integrate rocksplicator with exisiting application running on rocksdb.

I have a standalone application running on rocksdb. I want to make it distributed for hackathon. I don't want to replace rocksdb with any distributed DB so we decided to use rocksplicator.

Do we need to change our application to use rocksdplicator or it is separate process. We just need to point it to the DBs.

Does rocksplicator have rocksdb coloumn family support?

We already have rocksdb instances which uses column family. We want to replicate those using rocksplicator. Is it possible?

Unbale to build

In file included from /home/linuxpham/servers/temp/rocksplicator/common/stats/stats.cpp:20:
/usr/local/include/folly/stats/BucketedTimeSeries-defs.h:28:1: error: redefinition of ‘folly::BucketedTimeSeries<VT, CT>::BucketedTimeSeries(size_t, folly::BucketedTimeSeries<VT, CT>::Duration)’
BucketedTimeSeries<VT, CT>::BucketedTimeSeries(
^~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/local/include/folly/stats/BucketedTimeSeries.h:493,
from /usr/local/include/folly/stats/MultiLevelTimeSeries.h:25,
from /home/linuxpham/servers/temp/rocksplicator/common/stats/stats.h:72,
from /home/linuxpham/servers/temp/rocksplicator/common/stats/stats.cpp:16:
/usr/local/include/folly/stats/BucketedTimeSeries-inl.h:27:1: note: ‘folly::BucketedTimeSeries<VT, CT>::BucketedTimeSeries(size_t, folly::BucketedTimeSeries<VT, CT>::Duration)’ previously declared here

how to deal with data move

I check the code，find rebanlance only switch master with least_loaded_slave,no data is moved or deleted.
If disk will full,we need move some data to a new server,how to deal with this situation?

enable travis

@jparise mind enabling travis in https://travis-ci.org/pinterest/rocksplicator?

Why travis ci use thrift 9.1.0 while Dockerfile using 9.3.0

rocksplicator/.travis.yml

Line 63 in cd5cc34

- wget http://archive.apache.org/dist/thrift/0.9.1/thrift-0.9.1.tar.gz

BTW, why fbthrift and apache thrift coexist here. From the code, I only see the fbthrift functions.

Thanks.

Can offer a JAVA API document.

I am a java developer,it's far away for me to using C++ API.please help.

add me as collaborator

@jparise :)

Build with Bazel

Hi team 👋

Would you be open to building this repository with Bazel? It'd better integrate in our internal monorepo and I'd be happy to provide the implementation.

Abuout HotKeyDetector

I think HotKey Detector is an interesting class, but I don't understand in what scenarios this class is used?

unable to create cluster

Hi all,
im trying to get to know the rocksplicator setup. I configured the docker and trying to create a new cluster and im getting connection time out error.

 No handlers could be found for logger "kazoo.client"
 Traceback (most recent call last):
  File "rocksdb_admin.py", line 114, in <module>
     zk_client = _get_zk_client()
   File "rocksdb_admin.py", line 110, in _get_zk_client
     c.start()
   File "/usr/local/lib/python2.7/dist-packages/kazoo/client.py", line 567, in start
     raise self.handler.timeout_exception("Connection time-out")
 kazoo.handlers.threading.KazooTimeoutError: Connection time-out
 

i guess im missing something very basic.
can someone help me in this..?

request->need_rounter need to be set by set_need_routing(false) to prevent from infinite recursion

rocksplicator/examples/counter_service/counter_handler.cpp

Line 131 in 9849fc3

clients[0]->future_setCounter(*request).then(

otherwise reading an undefined key leads to infinite recursion.

Encryption At Rest

Is there any plan to implement Encryption At Rest?

is a ':' missing when setting class path at createVM?

https://github.com/pinterest/rocksplicator/blob/master/rocksdb_admin/helix_client.cpp#L46

Should it be ?
auto class_opt = "-Djava.class.path=" + class_path + "cluster_management-0.0.1-SNAPSHOT-jar-with-dependencies.jar:" + env_class_path;

Can rocksplicator support static compilation？

I added to the CMakeFiles file

set(CMAKE_EXE_LINKER_FLAGS "-static -static-libgcc")

But the compilation fails.

how to set hdfs_name_node?

we may want to use the active name node since backup needs write permission, while the hdfs_name_node is always hard-coded, and two name nodes can fail over to each other at any time.

how does pinterest make this work?

How long does it cost to change slave into a master?

Now, our own replication method of rocksdb costs about 20s to turn the slave into a master.

Host Docker image on Pinterest Docker account

We should probably move rajathprasad/rocksplicator-build:latest to something at the organization level like pinterest/rocksplicator-build:latest

how can ThriftRouter FileWatcher get the update from ConfigGenerator.java's post request?

https://github.com/pinterest/rocksplicator/blob/master/cluster_management/src/main/java/com/pinterest/rocksplicator/ConfigGenerator.java#L193 updates the latest layout to a http post request, while https://github.com/pinterest/rocksplicator/blob/master/common/thrift_router.h#L112 watches a local file. Does it mean there is a local http service that receives from the http post and saves it to a file thrift_router watches?

And ConfigGenerator is called by only the participant leader, how could other participants watch the file update...?

How to use the example (counter)

Hi, thanks for open-sourcing this great lib!

I am having problems running the example counter application. I've successfully built everything, but when I run the example I get the following error:

root@f693fedf78d4:/rocksplicator/build# ./examples/counter_service/counter 
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0423 12:47:09.583624  4238 availability_zone.cpp:69] Got invalid az: 
I0423 12:47:09.591229  4241 thrift_client_pool.h:151] Started counter::CounterAsyncClient thrift client IO thread
I0423 12:47:09.591681  4242 thrift_client_pool.h:151] Started counter::CounterAsyncClient thrift client IO thread
E0423 12:47:09.592037  4243 file_watcher.cpp:196] Failed to inotify_add_watch() with errno 2 : No such file or directory
F0423 12:47:09.592182  4238 thrift_router.h:118] Check failed: common::FileWatcher::Instance()->AddFile( config_path_, [this, local_group] (std::string content) { std::shared_ptr<const ClusterLayout> new_layout( parser_(std::move(content), local_group)); { folly::RWSpinLock::WriteHolder write_guard(layout_rwlock_); cluster_layout_.swap(new_layout); } }) Failed to watch 
*** Check failure stack trace: ***
Aborted

I guess I need to pass shard_config_path. What is the syntax of this file?

What happens when I run the example application? I understand it starts a server. I guess the python cluster management scripts communicate with this server? So is this a single Master node? How do I set up Slaves and how do I connect these components?

What is the advantage of using these admin tools / cluster management modules, as opposed to instantiating RocksDBReplicator directly from my code?

Thanks!

consistency question for replication

hi, @newpoo @shuzhang1989
i have a question for rocksplicator replication recently.
consider such case: if master node is down with the latest seq no for 100, and the latest seq no for slaves is 98, and the slave become master with latest seq no for 98. after a meanwhile, the master is up and become slave, and the new master's seq no is 101. according to seq no , the old master to be slave will pull data from the new master start with 100.

how can rocksplicator handle this case ?
Is the older master will hold diff data with the new master for seq no 99 and 100 ？
thank you !

Can rocksplicator bakup rocksdb to s3?

Can rocksplicator bakup rocksdb to s3?

For building only rocksdb_replicator from rocksplicator, Does it require both fbthrift and thrift

I am only building rocksdb_replicator and common modules from rocksplicator in my own debian environment. I installed both fbthrift and thrift according to docker file. I am getting undefined reference to

CMakeFiles/thrift_router_test.dir/thrift_router_test.cpp.o: In function apache::thrift::server::TConnectionContext::setHeader(std::string const&, std::string const&)': /usr/local/include/thrift/lib/cpp/server/TConnectionContext.h:77: undefined reference to apache::thrift::transport::THeader::setHeader(std::string const&, std::string const&)'
CMakeFiles/thrift_router_test.dir/thrift_router_test.cpp.o: In function apache::thrift::server::TConnectionContext::getHeader() const': /usr/local/include/thrift/lib/cpp/server/TConnectionContext.h:69: undefined reference to typeinfo for apache::thrift::transport::THeader'
/usr/local/include/thrift/lib/cpp/server/TConnectionContext.h:69: undefined reference to typeinfo for apache::thrift::transport::THeader' thrift/libdummy_service_thrift.a(DummyService.cpp.o): In function apache::thrift::HandlerCallbackBase::transform(folly::IOBufQueue&)':

Why use fbthrift，not use apache thrift?

Is it because you need to rely on any special function in fbthrift?
I plan to use rocksplication in my own project,
Can I use thrift instead of fbthrift?

Failed to apply updates to SLAVE /internal_db Invalid argument: Invalid column family specified in write batch

I have one master and one slave node. They are running rocksplicator. I am using only rocksdb_replicator module of this repo.

Once I start the master, I open the DB in master mode and add for replication. This DB has 10 column families.
While starting the slave, I open the DB in slave mode but after some time I got this error "Failed to apply updates to SLAVE internal_db Invalid argument: Invalid column family specified in write batch"
I can list all the column families using ldb and able to read 2 column families but the rest doesn't have any content.

What can be the issue here?

Basic test on replicating RocksDB - lack of documentation

Hey,
So I have a basic test that measures rates of insert and get keys from RocksDB, as part of internal tests to use RocksDB and rocksplicator for a project.
Now that I have the throughput for one RocksDB instance I'm trying to have another instance of RocksDB so we can increase the throughput.
I'm not sure what API should I use for it, I tried to follow the counter_service example, but there wasn't many comments on what does any of the part is doing. Nor which client should I use if I have a counter service up and running.
Hope you have some piece of information you could point me to.

Unable to build

Hi,

I am trying to build rocksplicator for the first time and am having trouble doing so. I am not a docker pro and was just running the commands mentioned in the README.
I pulled the docker image cdonaghy/rocksplicator-build and cloned the repo to ~/code/rocksplicator

Can you tell me if what should be present in $HOME/docker-root?
"Get into the docker build environment. We are assuming the rocksplicator repo is under $HOME/code/, and $HOME/docker-root is an existing directory."

docker run -v <SOURCE-DIR>:/rocksplicator -v $HOME/docker-root:/root -ti rocksplicator-build bash

What should be the value of <SOURCE-DIR>?

I just ran with

docker run -v ~/code/rocksplicator -v $HOME/docker-root:/root -ti rocksplicator-build bash

but it says

Unable to find image 'rocksplicator-build:latest' locally docker: Error response from daemon: repository rocksplicator-build not found: does not exist or no pull access.

$ docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
cdonaghy/rocksplicator-build   latest              fa0cbfa908de        7 weeks ago         2.48 GB

Thanks

Archiving rocksplicator project

Rocksplicator users and contributors,

Pinterest decided to archive rocksplicator project and will no longer be maintaining or updating the open source code.

If you have any thoughts or concerns regarding this, please comment on this issue.

Thank you

unable to set up replication

Hi, I’m trying to create a replication setup between two hosts I have. Its not clear how it works from the code/docs/whatsoever thats available about rocksplicator. I was successful in setting up the server, not sure what should be done next. How is data replicated between two machines? What are the further steps to be followed? Can someone give clarification? Thanks.