zeromicro/cds

Data syncing in golang for ClickHouse.

cds's Issues

请问源端的update是如何处理的?

是使用这种方式吗 https://mp.weixin.qq.com/s/a8OfsBn9VFnj7oxp0IIVGg
ReplacingMergeTree + 视图(argMax)?

ERROR-全量同步过程中出现错误

环境: centos 7 | clickhouse ClickHouse server version 20.12.5 | cds 最新版(建议添加一个版本号, commitid: 28c51c5)

现像: 在配置好全量同步,执行同步过程出现如下错误:

{"@timestamp":"2021-01-04T17:03:44.268+08","level":"error","content":"mysqltypeconv.go:59 sql: Scan error on column index 4, name \"Default\":
 converting NULL to string is unsupported"}

可能原因: 建表语句指定了字段类型, 导致NULL转换失败

问题:

* 建表语句是否可支持 Nullable
* 全库同步需要选中所有表 ?

请问该工具支持同步数据源的Update/Delete操作吗？

make up时候运行错误

Traceback (most recent call last):
File "/tmp/init_db.py", line 74, in
init_mongo()
File "/tmp/init_db.py", line 69, in init_mongo
collection.insert_many(result)
File "/usr/local/lib/python3.9/site-packages/pymongo/collection.py", line 761, in insert_many
blk.execute(write_concern, session=session)
File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 528, in execute
return self.execute_command(generator, write_concern, session)
File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 359, in execute_command
client._retry_with_session(
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
return self._retry_internal(retryable, func, session, bulk)
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1414, in _retry_internal
raise last_error
File "/usr/local/lib/python3.9/site-packages/pymongo/mongo_client.py", line 1416, in _retry_internal
return func(session, sock_info, retryable)
File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 353, in retryable_bulk
self._execute_command(
File "/usr/local/lib/python3.9/site-packages/pymongo/bulk.py", line 309, in _execute_command
result, to_send = bwc.execute(ops, client)
File "/usr/local/lib/python3.9/site-packages/pymongo/message.py", line 907, in execute
result = self.write_command(request_id, msg, to_send)
File "/usr/local/lib/python3.9/site-packages/pymongo/message.py", line 999, in write_command
reply = self.sock_info.write_command(request_id, msg)
File "/usr/local/lib/python3.9/site-packages/pymongo/pool.py", line 771, in write_command
helpers._check_command_response(result, self.max_wire_version)
File "/usr/local/lib/python3.9/site-packages/pymongo/helpers.py", line 151, in _check_command_response
raise NotMasterError(errmsg, response)
pymongo.errors.NotMasterError: operation was interrupted, full error: {'errorLabels': ['RetryableWriteError'], 'topologyVersion': {'processId': ObjectId('6071d9eecc09c5721c8f196f'), 'counter': 6}, 'operationTime': Timestamp(1618074848, 500), 'ok': 0.0, 'errmsg': 'operation was interrupted', 'code': 11602, 'codeName': 'InterruptedDueToReplStateChange', '$clusterTime': {'clusterTime': Timestamp(1618074848, 500), 'signature': {'hash': b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'keyId': 0}}}
make: *** [init] Error 1

make up mongo error

mongodb test_mongo.example inserted 10000 lines
Traceback (most recent call last):
File "/tmp/init_db.py", line 74, in
init_mongo()
File "/tmp/init_db.py", line 69, in init_mongo
collection.insert_many(result)
File "/usr/local/lib/python3.10/site-packages/pymongo/collection.py", line 615, in insert_many
blk.execute(write_concern, session=session)
File "/usr/local/lib/python3.10/site-packages/pymongo/bulk.py", line 459, in execute
return self.execute_command(generator, write_concern, session)
File "/usr/local/lib/python3.10/site-packages/pymongo/bulk.py", line 351, in execute_command
with client._tmp_session(session) as s:
File "/usr/local/lib/python3.10/contextlib.py", line 135, in enter
return next(self.gen)
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1656, in _tmp_session
s = self._ensure_session(session)
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1643, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1594, in __start_session
server_session = self._get_server_session()
File "/usr/local/lib/python3.10/site-packages/pymongo/mongo_client.py", line 1629, in _get_server_session
return self._topology.get_server_session()
File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 534, in get_server_session
session_timeout = self._check_session_support()
File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 520, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.10/site-packages/pymongo/topology.py", line 223, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: mongo1:30001: [Errno -2] Name or service not known,mongo2:30002: [Errno -2] Name or service not known,mongo3:30003: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 624a83e9fd437d56bbaaf7a3, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('mongo1', 30001) server_type: Unknown, rtt: None, error=AutoReconnect('mongo1:30001: [Errno -2] Name or service not known')>, <ServerDescription ('mongo2', 30002) server_type: Unknown, rtt: None, error=AutoReconnect('mongo2:30002: [Errno -2] Name or service not known')>, <ServerDescription ('mongo3', 30003) server_type: Unknown, rtt: None, error=AutoReconnect('mongo3:30003: [Errno -2] Name or service not known')>]>
make: *** [init] Error 1

mac quickstart

ERROR: for canal-server Container "4252636c3eef" is unhealthy.
ERROR: Encountered errors while bringing up the project.

是否会考虑支持ReplicatedCollapsingMergeTree？

开发组成员们，你们好。
在研究本项目的时候产生了一个疑问，希望能得到解答。
背景：在Galaxy生成的Clichouse DDL中，为每个表建立了”ck_is_delete“列，复制表的引擎选用：ReplicatedMergeTree。
问题：这里为什么没有选用ReplicatedCollapsingMergeTree作为复制表的引擎，使用”sign“代替”ck_is_delete“？

望解答

make up 报错

make up 报错

请问支持创建分布式表吗

我这边clickhouse是一个3分片集群, 配置同步任务时clickhouse这边创建的是分布式表还是普通单表啊?

请问是否有源端和目标端的数据校验机制?

RT

亿级表全量同步效率如何

同步的过程中遇到这个问题 Scan error on column index 4, name \"Default\": converting NULL to string is unsupported"}

mysql 的 binlog 实时解析,没有看到,你们这个同步工具,有没有做mysql binlog实时同步??

mysql 的 binlog 实时解析,没有看到,你们这个同步工具,有没有做mysql binlog实时同步??

我全文搜索了 binlog ,没有发现binlog这个插件,请问你们实现了binlog解析吗?

微信群，好像过期了，是否可以更新？

如题！

正在研究Go大数据相关生态，想加入社群，了解更多。

希望能及时更新微信二维码，谢谢！

MySQL灾备

这个可以作为MySQL异地主从备份吗？当另一MySQL故障时，切到当前MySQL

完全按照 quickstart， 最后应用层安装出现找不到目录问题

完全按照 quickstart， 最后应用层安装出现找不到目录问题