Mongodb 3.4 版本下不会同步索引

[alibaba]

Mongodb 3.4 版本下不会同步索引

[vinllen]

你的问题是没有同步索引？dml_only配置项是否启用？

[wangyiyang]

replayer.dml_only = true

[vinllen]

设置成false 过一遍wiki的文档

[wangyiyang]

@vinllen 我改成false重新尝试了下，步骤如下：

1. 启动mongo-shake，链接数据库A,B
2. A数据库，插入文档，collections之前不存在
3. A数据库，创建索引
4. 在B中查看索引
   结果发现索引并没有同步过来，配置如下：

# connect source mongodb, set username and password if enable authority.
# split by comma(,) if use multiple instance in one replica-set.
# split by semicolon(;) if sharding enable.
mongo_urls = mongodb://10.121.9.120:30012

# collector name
collector.id = mongoshake

# save checkpoint every ${batch_size}. reduce the number
# of batch_size will saving the checkpoint with higher frequency
# default size is 16 * 1024
checkpoint.interval = 5000

# http api interface. Users can use this api to monitor mongoshake.
# We also provide a restful tool named "mongoshake-stat" to 
# print ack, lsn, checkpoint and qps information based on this api.
# usage: `./mongoshake-stat --port=9100`
http_profile = 9100
# profiling on net/http/profile
system_profile = 9200


# global log level. lower level message
# will be filter
log_level = info
# log file name. use logs/ prefix folder path
log_file = collector.log
# log buffer or unbuffer. If set true, logs may not be print when exit. If
# set false, performance will be decreased extremely
log_buffer = true


# filter db/collection namespace. at most one of these two parameters can be given.
# if the filter.namespace.black is not empty, the given namespace will be
# filtered while others namespace passed. 
# if the filter.namespace.white is not empty, the given namespace will be
# passed while others filtered. 
# all the namespace will be passed if no condition given.
# split by semicolon(;)
filter.namespace.black =
filter.namespace.white =

# this parameter is not supported in current opensouce version.
# oplog namespace and global id. others oplog in 
# mongo cluster that has distinct global id will 
# be discard. Query without gid (means getting all
# oplog out) if no oplog.gid set
oplog.gids =   


# [auto] 		decide by if there has uniq index in collections. 
# 		 		use `collection` with uniqu index set otherwise `id`
# [id] 			shard by ObjectId. handle oplogs in sequence by unique _id 
# [collection] 	shard by ns. handle oplogs in sequence by unique ns 
shard_key = auto


# syncer send time interval, unit is second.
# flush the syncer reader buffer time interval.
syncer.reader.buffer_time = 3


# oplog transmit worker concurrent
worker = 8
worker.batch_queue_size = 64

# batched oplogs have block level checksum value using 
# crc32 algorithm. and compressor for compressing content
# of oplog entry. 
# supported compressor are : gzip,zlib,deflate
# Do not enable this option when tunnel type is "direct"
worker.oplog_compressor = none


# tunnel pipeline type. now we support rpc,file,kafka,mock,direct
tunnel = direct
# tunnel target resource url
# for rpc. this is remote receiver socket address
# for tcp. this is remote receiver socket address
# for file. this is the file path, for instance "data"
# for kafka. this is the topic and brokers address which split by comma, for
# instance: topic@brokers1,brokers2, default topic is "mongoshake"
# for mock. this is uesless
# for direct. this is target mongodb address
tunnel.address = mongodb://10.121.9.120:30088



# collector context storage mainly including store checkpoint
# type include : database, api
# for api storage, address is http url
# for database storage, address is collection name while db name is "mongoshake" by default.
# if source database is replica, context will store in the source mongodb.
# if source database is sharding, context must be given to point the config server address.
# context.storage.url is only used in sharding. When source mongodb type is replicaSet, 
# checkpoint will write into source mongodb as default.
context.storage = database
#context.storage.url = mongodb://127.0.0.1:20070
context.address = ckpt_default
# pay attention: this is UTC time which is 8 hours latter than CST time. this 
# variable will only be used when checkpoint is not exist.
context.start_position = 2000-01-01T00:00:01Z


# high availabity option.
# enable master election if set true. only one mongoshake can become master
# and do sync, the others will wait and at most one of them become master once 
# previous master die. The master information stores in the `mongoshake` db in the source 
# database by default.
# This option is useless when there is only one mongoshake running.
master_quorum = false


# ----------------------splitter----------------------
# if tunnel type is direct, all the below variable should be set

# only transfer oplog commands for syncing. represent
# by oplog.op are "i","d","u". also include applyOps
replayer.dml_only = false

# executors in single worker
replayer.executor = 1

# oplog changes to Insert while Update found non-exist (_id or unique-index)
replayer.executor.upsert = true
# oplog changes to Update while Insert found duplicated key (_id or unique-index)
replayer.executor.insert_on_dup_update = true
# db. write duplicated logs to mongoshake_conflict
# sdk. write duplicated logs to sdk.
replayer.conflict_write_to = none

# replayer duration mode. drop oplogs and take
# no any action(only for debugging enviroment) if 
# set to false. otherwise write to ${mongo_url} instance
replayer.durable = true

[wangyiyang]

@vinllen 我又确认了下 mongo的版本是4.0.1