Comments (19)
Please be careful if you want to build a two-way sync tunnel, the data may be moved from the source to the target, and then from the target to the source again.
Check out this question(How to build active-active replication(双活) in the current opensource version without gid support?) in the faq to break the loop.
from mongoshake.
- No. If you use MongoShake in two-way sync, the consistency can't be guaranteed.
- 9100 port is only used when running incr-sync stage, so if your MongoShake is running full-sync stage, you should use 9101 port, however, mongoshake-stat doesn't support 9101. You can curl the 9101 port directly to get the monitor data by:
curl 127.0.0.1:9101
.
from mongoshake.
Currently, we don't have the tutorial. You can check out the usage to start MongoShake, if you want to set up with detail information, read the comment of configuration and the wiki faq
from mongoshake.
O.K. when I have a few questions about it:
- How do I run the packages file on
mongo-shake/src/vendor
? - Is there also a script to install govendor and is it correct to set the path to
GOPATH=pwd/../..;
? - Is there an example config for a two way sync of mongodb between two instances? I'm looking at the two config files and I'm having a hard time figuring out which options I need to modify.
from mongoshake.
- You should install
govendor
first and then use this to download all dependency packages which locate onmongo-shake/src/vendor
. govendor
is an opensource tool,GOPATH
should be set before using. Generally speaking, this should point to the MongoShake root path.- Currently, no. You can try to build one-way sync first, after that, two-way sync is simple.
from mongoshake.
I read the How to build active-active replication in the FAQ but it sounds like that active-active is not supported since I need to define a one way sync on collection level. Or do I miss something?
from mongoshake.
Yes, the granularity of two-way syncing is collection level, so that you should filter some database/collection on one side, and filter the other db/collections on the other side.
It is because mongodb doesn't support gid, so it's the only way to prevent data replication from forming a loop.
from mongoshake.
So basically that means I can't keep a collection in sync if I make changes on both servers to this collection?
from mongoshake.
yes
from mongoshake.
That's unfortunate. Then I need to look for another solution. Thank you for the clarification.
from mongoshake.
That's unfortunate. Then I need to look for another solution. Thank you for the clarification.
Hi Jwillmer, Did you find solution for a two way sync of mongodb between two instances?
from mongoshake.
That's unfortunate. Then I need to look for another solution. Thank you for the clarification.
Hi Jwillmer, Did you find solution for a two way sync of mongodb between two instances?
We plan to build an app that connects to a remote instance, listens to the events and publishes them in the local instance. This gives us also the benefit to filter what we like to keep in sync.
from mongoshake.
That's unfortunate. Then I need to look for another solution. Thank you for the clarification.
Hi Jwillmer, Did you find solution for a two way sync of mongodb between two instances?
We plan to build an app that connects to a remote instance, listens to the events and publishes them in the local instance. This gives us also the benefit to filter what we like to keep in sync.
So you are developing the sync app, right?
Have you checked https://github.com/yougov/mongo-connector, our team prefers to use mongo-connector, but I need to figure out the feasibility of these tools.
I would appreciate it if you could give any advise.
from mongoshake.
I would appreciate it if you could give any advise.
We have the development of the tool in the backlog. So far I can't help u with expertise. Depending on the prioritization I can answer your questions in a couple of weeks.
from mongoshake.
I would appreciate it if you could give any advise.
We have the development of the tool in the backlog. So far I can't help u with expertise. Depending on the prioritization I can answer your questions in a couple of weeks.
OK I got it. Thanks a lot.
from mongoshake.
hi @vinllen, I am start using MongoShake. can I have some question about lastest version release-v2.4.9-20200806:
- Is this version support two-way sync tunnel without using GID? I have 2 replicaset clusters, each have 3 node: one primary and two secondary, and 2 MongoShake to sync data between them. I write data to one cluster in one time (active-standby) and I saw that MongoShake run properly.
- how to use mongoshake-stat to monitor? I using command: ./mongoshake-stat --port=9100, but failed with error: ValueError: No JSON object could be decoded.
Thanks a lot.
from mongoshake.
- No. If you use MongoShake in two-way sync, the consistency can't be guaranteed.
- 9100 port is only used when running incr-sync stage, so if your MongoShake is running full-sync stage, you should use 9101 port, however, mongoshake-stat doesn't support 9101. You can curl the 9101 port directly to get the monitor data by:
curl 127.0.0.1:9101
.
- Note that I write data to only one cluster in one time, mean active-standby, not active-active. Assuming that I have 2 clusters A and B, mongoshake 1 replicates data from A to B and mongoshake 2 replicates data from B to A. I insert 10 document to A, then B has 10 document replicated from A by mongoshake 1, and no data replicated back from B to A by mongoshake 2. No infinite data replication loop occur as you mention in GID feature. So why consistency can't be guaranteed? Do I miss something? Please explain to me.
from mongoshake.
You can't break the sync loop without GID supported. Checkout this scenario, I just update {_id:1, a:1}
to {_id:1, a:2}
and generate an oplog: {o2:{_id:1}, o:{_id:1, a:2}}
, when this oplog sync from A to the B, the data with {_id:1}
will be updated and generate another oplog: {o2:{_id:1}, o:{_id:1, a:2}}
. So once this oplog synced back to A, the {_id:1}
will be updated again even the result is not changed. So there will be a loop: A->B->A->B->...
As a result, when you update {_id:1}
again to {_id:1, a:3}
, the result of {_id:1}
on A and B will switch back and forth between {a:2}
and {a:3}
.
from mongoshake.
Please open a new issue if anyone has a question, this issue should be frozen.
from mongoshake.
Related Issues (20)
- mongodb 3.2.10使用mongo-shake 2.6.6往kafka同步,启动任务报#topic@ip:port] create writer error[address format error]
- 单实例之间同步no oplog HOT 1
- Add MongoShake Prometheus Exporter to Monitoring HOT 1
- mongoshake-stat 执行报错
- Database sharding support for incremental sync
- 源端是单实例 HOT 1
- 源端压测时,单表同步写入性能瓶颈求助
- 源端单实例像目标端分片同步数据报错
- mongoshake 同步状态某个集合卡住不动了
- 全量+增量同步,差不多全量同步完异常退出 HOT 1
- run replication failed: select sync mode failed: get all timestamp failed: (Unauthorized) command find requires authentication
- mongo 到kafka ,开启全字段同步后,会导致mongos 挂掉,如果是关掉就没有问题
- Does it support mongodb-6.0?
- 如何进行mongo的冷数据备份?哪位有成熟的方案?
- mongodb 3.2 单节点,开启 oplog, 同步到4.4,全量加增量模式,已关闭ddl操作,但是因为ddl同步抱错
- 数据冷热备份方案
- 同步集群后发现缺少sensitive_event集合,但不清楚什么原因造成的
- Mongoshake的集群到集群单向同步是否支持反转同步方向
- 生产环境中使用,如何关闭用于查看内部go堆栈profiling端口
- Mongoshake同步时写入失败
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mongoshake.