qihoo360 / mongosync Goto Github PK

mongosync is simple && useful tool to sync data between mongo replicaSet, it is developed by Qihoo's DBA and infrastructure team

Makefile 0.06% C++ 93.73% Python 4.87% Perl 0.24% C 1.10%

mongosync's Introduction

README

中文wiki前往

1.Introduction

This simpe program is for transfering data between two mongo nodes, including:

data cloning;
oplog syncing(applying one mongo databbase's oplog to another mongo database) ;
oplog cloning(not applying, just storing the oplogs from source mongo database to another one)

2.Usage

Get the source code

$ git clone https://github.com/Qihoo360/mongosync.git

Compile

Enter the source code and typewrite:
```
$ make
```
after that, a "mongosync" named execution file appears in the $(MONGOSYNC_ROOT)/output directory, and also a "mongosync.conf" named config file

SP:

a. maybe some depenncies not installed in your machine, install them according to the tips（mostly sons, boost and ssl is needed）

b. don't use c++11 to compile this program, otherwise will occur error

Exectution

The program can launches with command line or config file，and the specified usage can be obtained by:

$ cd output
$ ./mongosync --help
Follow is the mongosync-surpported options:
--help                   to get the help message
-c conf.file             use config file to start mongosync
--src_srv arg            the source mongodb server's ip port
--src_user arg           the source mongodb server's logging user
--src_passwd arg         the source mongodb server's logging password
--src_auth_db arg        the source mongodb server's auth db
--src_use_mcr            force source connection to use MONGODB-CR password machenism
--dst_srv arg            the destination mongodb server's ip port
--dst_user arg           the destination mongodb server's logging user
--dst_passwd arg         the destination mongodb server's logging password
--dst_auth_db arg        the destination mongodb server's auth db
--dst_use_mcr            force destination connection to use MONGODB-CR password machenism
--db arg                 the source database to be cloned
--dst_db arg             the destination database
--coll arg               the source collection to be cloned
--dst_coll arg           the destination collection
--oplog                  whether to sync oplog
--raw_oplog              whether to only clone oplog
--op_start arg           the start timestamp to sync oplog
--op_end arg             the start timestamp to sync oplog
--dst_op_ns arg          the destination namespace for raw oplog mode
--no_index               whether to clone the db or collection corresponding index
--filter arg             the bson format string used to filter the records to be transfered

3.Performance

Rough testing under the situation:

Ping from this program's exectution's machine to the mongo db server's machine(two machines) is about 1.5ms；
total about 36,400,000 documents；
database size about 1.09GB

transfering time is about 379 seconds(even qps: 96.04k).

SP: The bottleneck is writing, So using multiple threads(multi-conn) to accelerate the speed of that(but also limited by database server's writing speed).

mongosync's People

Contributors

Stargazers

Watchers

Forkers

hongyu01 lengyun123456 dlpc satchel9 24 liuguoshun fengshao0907 chinayin coolzhang garydot callmedba coneboy-k running-ball dumingyou aricy wenzhenghu cyang23 cuiopen jellyabd bridge19 qyt2018 skyformat99 zhaoshiling1017 kssssssssss yuanddqiao markzhou123 yanghongkjxy shiqml harry775 jiangnanora wordgod123 soulssh weibuzudao qianhua noel1992 xjtugong isgasho zhonghua001 gaoxw126 xidianwlc shiftwinting thinking-one-hour-everyday dongjiashun theharveyz cxlan syniguo cnxxpp wanglibao8 jason0bourne lilianxiang dbsky1985 liubeibeiliu outshow kevonz

mongosync's Issues

编译不成功

[root@h0068020 mongosync-master]# make
scons -C ./dep/mongo-cxx-driver --ssl install -j 12
scons: Entering directory `/root/mongosync-master/dep/mongo-cxx-driver'
scons: Reading SConscript files ...
File "/root/mongosync-master/dep/mongo-cxx-driver/SConstruct", line 507

print env_vars.GenerateHelpText(env)

             ^

SyntaxError: invalid syntax

make: *** [dep/mongo-cxx-driver/build/install/lib/libmongoclient.a] Error 2

centos 6.5中，make不过呐

Checking for libstdc++ 4.6.0 or better... no
Checking if we are on a POSIX system... yes
Checking if the POSIX monotonic clock is supported... yes
Checking for __thread... no
Checking for gcc __atomic builtins... no
Checking for useable __sync builtins... no
Checking for C++ header file boost/version.hpp... no
Could not find boost headers in include search path
make: *** [dep/mongo-cxx-driver/build/install/lib/libmongoclient.a] Error 1

是不是依赖的版本太低呐？能不能列出安装的依赖包版本，谢谢
gcc-c++ 4.4.7
python 2.6.6
scons 2.0.1

同步的时候直接导致 mongo 从库挂掉了

[INFO ] May 16 17:38:47 2017 [mongosync] cloning xxx.contacts to xxx.contacts
[INFO ] May 16 17:38:47 2017 [mongosync] cloning xxx.contacts indexes
[INFO ] May 16 17:38:47 2017 [mongosync] clone xxx.contacts indexes success, total 5objects

[WARN ] exception occurs: xxx:2009910278 dbclient error communicating with server: xxx:20099, retry it
terminate called after throwing an instance of 'mongo::SocketException'
what(): socket exception [FAILED_STATE] for xxx:20099 (xxx.31) failed

同步造成源库进程死掉

数据同步过程中，执行大概10分钟之后，源库进程死掉。

mongodb版本号：3.2.3 64位
系统版本号：centos 7.3

mongosync 执行同步报错

@baotiao
@gaodq
@smartOf360
@flabby

[root@hadoopdn-02 mongosync_1_0]# ./ mongosync --src_srv 192.168.4.24:10011 --src_user root --src_passwd xxxx --src_auth_db admin --db src --dst_db dst--dst_srv 192.168.4.32:10011 --dst_user root --dst_passwd 12345678 --dst_auth_db admin
[INFO ] monogosync started, first set log level to INFO
[INFO ] May 24 17:23:20 2017 [mongosync] connect to srv_rsv: 192.168.4.24:10011 ok!
[INFO ] May 24 17:23:20 2017 [mongosync] srv: 192.168.4.24:10011, dbname: admin ok!
[INFO ] May 24 17:23:20 2017 [mongosync] connect to srv_rsv: 192.168.4.32:10011 ok!
[INFO ] May 24 17:23:20 2017 [mongosync] srv: 192.168.4.32:10011, dbname: admin ok!
[INFO ] May 24 17:23:20 2017 [mongosync] cloning db: src----> dst

[INFO ] May 24 17:23:20 2017 [mongosync] cloning src.groupTest1 to dst.groupTest1
[WARN ] exception occurs: 10278 dbclient error communicating with server: 192.168.4.24:10011, retry it
terminate called after throwing an instance of 'mongo::SocketException'
what(): socket exception [FAILED_STATE] for 192.168.4.24:10011 (192.168.4.24) failed
Aborted (core dumped)
[root@hadoopdn-02 mongosync_1_0]# ./mongosync
[INFO ] monogosync started, first set log level to INFO
[FATAL] Wrong argument number
Follow is the mongosync-surpported options:
--help to get the help message
-c conf.file use config file to start mongosync
--src_srv arg the source mongodb server's ip port
--src_user arg the source mongodb server's logging user
--src_passwd arg the source mongodb server's logging password
--src_auth_db arg the source mongodb server's auth db
--src_use_mcr force source connection to use MONGODB-CR password machenism
--dst_srv arg the destination mongodb server's ip port
--dst_user arg the destination mongodb server's logging user
--dst_passwd arg the destination mongodb server's logging password
--dst_auth_db arg the destination mongodb server's auth db
--dst_use_mcr force destination connection to use MONGODB-CR password machenism
--db arg the source database to be cloned
--dst_db arg the destination database
--coll arg the source collection to be cloned
--dst_coll arg the destination collection
--oplog whether to sync oplog
--raw_oplog whether to only clone oplog
--op_start arg the start timestamp to sync oplog
--op_end arg the start timestamp to sync oplog
--dst_op_ns arg the destination namespace for raw oplog mode
--no_index whether to clone the db or collection corresponding index
--filter arg the bson format string used to filter the records to be transfered
--bg_num arg the background thread number for cloning data(not oplog syncing and oplog storing)
--batch_size arg the data grouping size criterion in cloning data(0-16M, default to 16M), unit is Byte
--log_level arg specify the log level
[root@hadoopdn-02 mongosync_1_0]#

每次启动都要做一次复制集的primary选举切换，然后就报错了，有遇到过吗？什么问题

编译通不过

Checking if the POSIX monotonic clock is supported... yes
Checking for __thread... no
Checking for gcc __atomic builtins... no
Checking for useable __sync builtins... no
Checking for C++ header file boost/version.hpp... no
Could not find boost headers in include search path
make: *** [dep/mongo-cxx-driver/build/install/lib/libmongoclient.a] Error 1
[root@10-9-161-112 mongosync]# gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[root@10-9-161-112 mongosync]# python
Python 2.7.5 (default, Nov 20 2015, 02:00:19)

现在支持从mongos和shard读入吗？

是否支持从mongos和shard读入
2 是否支持异构：比如从副本集的secondary读入，写入到mongos
3 全量同步现在是在同步全量数据库后，再追oplog同步吗？

Could not find boost headers in include search path

[root@svr mongosync]# make
scons -C ./dep/mongo-cxx-driver --ssl install -j 8
scons: Entering directory `/root/mongosync/mongosync/dep/mongo-cxx-driver'
scons: Reading SConscript files ...
Mkdir("build/scons")
scons version: 2.5.1
python version: 2 6 6 'final' 0
Checking whether the C++ compiler works... yes
Checking whether the C compiler works... yes
Checking if C++ compiler "g++" is GCC... yes
Checking if C compiler "gcc" is GCC... yes
Checking if C compiler supports -Wno-unused-local-typedefs... no
Checking if C compiler supports -Wno-unused-function... yes
Checking if C compiler supports -Wno-unused-private-field... no
Checking if C compiler supports -Wno-deprecated-declarations... yes
Checking if C compiler supports -Wno-tautological-constant-out-of-range-compare... no
Checking if C compiler supports -Wno-unused-const-variable... no
Checking if C compiler supports -Wno-missing-braces... yes
Checking if we are using libstdc++... yes
Checking for libstdc++ 4.6.0 or better... no
Checking if we are on a POSIX system... yes
Checking if the POSIX monotonic clock is supported... yes
Checking for __thread... yes
Checking for gcc __atomic builtins... no
Checking for useable __sync builtins... yes
Checking for C++ header file boost/version.hpp... no
Could not find boost headers in include search path
make: *** [dep/mongo-cxx-driver/build/install/lib/libmongoclient.a] Error 1

can't use 'local' database through mongos

terminate called after throwing an instance of 'mongo::UserException'
what(): nextSafe(): { $err: "can't use 'local' database through mongos", code: 13644 }

无法同步数据

从replicaSet是同步到shard，发现只能同步index，不能同步documents

Ubuntu 14 下编译不成功。

这是make的报错：

这是config.log里面的报错：

目的副本集的master切换后，报Aborted (core dumped)错误后退出

./mongosync --src_srv 192.168.167.1:20000 --dst_srv localhost:20000 --db data --dst_db data --colls credentials,users --bg_num 10 --oplog 2>log3

在使用以上语句进行集合模式实时同步时，目的副本集发生主从切换后，如果源端数据进行删除操作，mongosync就会挂掉，只有 Aborted (core dumped) 这一句报错。即使是源端数据进行插入操作，虽然mongosync不会挂掉，但是此操作并不能sync过来。

请问是mongosync不支持这种情况吗？

这个怎么解决

In file included from src/mongo/client/bulk_operation_builder.cpp:22:
src/mongo/client/dbclientinterface.h: In member function 'virtual void mongo::DBConnector::checkResponse(const char*, int, bool*, std::__cxx11::string*)':
src/mongo/client/dbclientinterface.h:598:13: error: this 'if' clause does not guard... [-Werror=misleading-indentation]
if( retry ) *retry = false; if( targetHost ) *targetHost = "";
^~
src/mongo/client/dbclientinterface.h:598:41: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
if( retry ) *retry = false; if( targetHost ) *targetHost = "";
^~
cc1plus: all warnings being treated as errors
scons: *** [build/linux2/ssl/mongo/client/bulk_operation_builder.o] Error 1
scons: building terminated because of errors.
make: *** [Makefile:60：dep/mongo-cxx-driver/build/install/lib/libmongoclient.a] 错误 2

Support versions?

This is some how very interesting according to docs mongosync just supports version 6 of mongoDB clusters... why it can not be used for older versions?

macOS sierra下载下来make错误

hi,请问我按照wiki上的步骤下载下来后进行make,出现了17个warning和1个error,error如下：
mongosync.cc:587:3: error: no matching function for call to 'time'
time(&cur);
^~~~
我试过直接下载编译好的二进制文件，但是无法运行，应该是编译环境不同的关系

那请问这个time相关的error要怎么处理呢？
error.txt

这个支持3.0以上的mongodb 同步么

mongosync这个支持3.0以上的mongodb 同步么

sync之后，没有迁移过去

相关log提示已经成功，但相关的collection没有迁移过去

背景：

从一个单点的数据集合迁移到一个shared server上

关于编译错误的问题

感觉文档中写的太不清楚了，其实在编译mongosync的时候会需要mongo c++驱动，所有先要编译c++驱动了；因此有时编译不同过很可能是驱动编译失败了。
错误日志位置：
mongosync/dep/mongo-cxx-driver/build/scons/config.log

make error！

scons -C ./dep/mongo-cxx-driver --ssl install -j 1
scons: Entering directory `/home/mike/mongosync/dep/mongo-cxx-driver'
scons: Reading SConscript files ...
scons version: 2.0.1
python version: 2 6 6 'final' 0
Checking whether the C++ compiler worksyes
Checking whether the C compiler worksyes
Checking if C++ compiler "g++" is GCC... yes
Checking if C compiler "gcc" is GCC... yes
Checking if C compiler supports -Wno-unused-local-typedefs... no
Checking if C compiler supports -Wno-unused-function... yes
Checking if C compiler supports -Wno-unused-private-field... no
Checking if C compiler supports -Wno-deprecated-declarations... yes
Checking if C compiler supports -Wno-tautological-constant-out-of-range-compare... no
Checking if C compiler supports -Wno-unused-const-variable... no
Checking if C compiler supports -Wno-missing-braces... yes
Checking if we are using libstdc++... yes
Checking for libstdc++ 4.6.0 or better... no
Checking if we are on a POSIX system... yes
Checking if the POSIX monotonic clock is supported... yes
Checking for __thread... no
Checking for gcc __atomic builtins... no
Checking for useable __sync builtins... no
Checking for C++ header file boost/version.hpp... yes
Checking for C++ library boost_regex-mt... no
Checking for C++ library boost_regex... no
can't find boost
make: *** [dep/mongo-cxx-driver/build/install/lib/libmongoclient.a] Error 1