ucargroup / datalink Goto Github PK
View Code? Open in Web Editor NEWDataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步,分布式、可扩展的数据交换平台。
License: Apache License 2.0
DataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步,分布式、可扩展的数据交换平台。
License: Apache License 2.0
想问一下关于sqlserver-reader的计划,大概的时间节点
现在的项目是否不支持从oracle中读取数据?
然后是否有计划支持?
如果可以共建的话,怎样加入?
个人想学习交流
reader是mysql,write是kafka,运行中报如下错误以及消费kafka是乱码
1、运行时候报错,出现如下错误(已解决)
Caused by: java.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of com/ucar/datalink/worker/core/util/classloader/RelPluginClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature
解决办法:kafka的plugin的lib依赖去除sl4j依赖包即可
2、kafka消息为乱码数据,如下片段
{"databaseName":"test","dbTableRowCellVOList":[{"afterValue":"啊啊啊啊啊啊","beforeValue":"中文啊啊啊","columnName":"name"},{"afterValue":"1","columnName":"id"}],"eventType":"UPDATE","id":"1","tableName":"test1"} ��*����� �| 3����mysql-bin.000008���� �*�UTF-80�ڠ��-8�B J PVX����C��Z?UPDATE
test1SET
name='啊啊啊啊啊啊' WHERE (
id='1')��� L����mysql-bin.000008����� �*�UTF-80�ڠ��-8�B�testJ�test1PYX�b� rowsCount��1������I��P b�� &� �������������id �( 0 B�1R bigint(20) /�����name ( 0 B�中文啊啊啊Rvarchar(100)�&� �������������id �( 0 B�1R bigint(20)�2�����name (�0 B�啊啊啊啊啊啊Rvarchar(100)�@ 1����mysql-bin.000008�Е�� �*�UTF-80�ڠ��-8�B J P���� ��8104191 ��*����� ��� 4����mysql-bin.000008����� �*�UTF-80�����-8�B J P��X����r��ZnINSERT INTO t_dl_task_delaytime (task_id,delay_time,create_time) VALUES (1,633, now())��� ^����mysql-bin.000008���� �*�UTF-80�����-8�B�datalinkJ�t_dl_task_delaytimeP?X�b� rowsCount��1������7��P b���/� �������������id �(�0 B�6R�BIGINT(20) unsigned�0�������������� delay_time (�0 B�633R BIGINT(20)�6���]��create_time (�0 B�2020-01-17 10:14:59R�DATETIME�+���������������task_id (�0 B�1R BIGINT(20)��� 4����mysql-bin.000008���� �*�UTF-80�����-8�B J P��X��������Z��INSERT INTO t_dl_task_statistic (task_id, records_per_minute, size_per_minute, write_time_per_record, exceptions_per_minute, read_write_count_per_minute, create_time) VALUES (1, 2, 398, 4.00, 0, 2, now())��� ^����mysql-bin.000008����� �*�UTF-80�����-8�B�datalinkJ�t_dl_task_statisticP
X�b�
rowsCount��1������;��P b���1� �������������id �(�0 B�970R�BIGINT(20) unsigned�+���������������task_id (�0 B�1R
BIGINT(20)�6���������������records_per_minute (�0 B�2R
BIGINT(20)�5���������������size_per_minute (�0 B�398R
DECIMAL(20,2)�9���������������exceptions_per_minute (�0 B�0R
BIGINT(20)�6���]��create_time (�0 B�2020-01-17 10:14:59R�DATETIME�?���������������read_write_count_per_minute (�0 B�2R
BIGINT(20)���
4����mysql-bin.000008�ܜ�� ��UTF-80�����-8�B J P��X��������Z��INSERT INTO t_dl_worker_jvm_state
(worker_id, host, old_mem_used, old_mem_max, young_mem_max, young_mem_used, create_time, interval_old_collection_count, interval_young_collection_count, interval_old_collection_time, interval_young_collection_time, current_thread_count)
VALUE
(2, '10.104.17.192', 0, 4294967296, 1610612736, 611795072, now(), 0, 0, 0, 0, 50)���
a����mysql-bin.000008�Š�� ��UTF-80�����-8�B�datalinkJ�t_dl_worker_jvm_stateP��X�b�
rowsCount��1������F��P b���1� �������������id �(�0 B�978R�BIGINT(20) unsigned�-�������������� worker_id (�0 B�2R
10.104.17.192R�VARCHAR(20)�0��������������old_mem_used (�0 B�0R
BIGINT(20)�8���������������old_mem_max (�0 B
4294967296R
young_mem_max (�0 B
1610612736R
BIGINT(20)�:���������������young_mem_used (�0 B 611795072R
BIGINT(20)�7���]��create_time (�0 B�2020-01-17 10:14:59R TIMESTAMP�A���������������interval_old_collection_count (�0 B�0R
BIGINT(20)�C� �������������interval_young_collection_count (�0 B�0R
BIGINT(20)�@�
�������������interval_old_collection_time (�0 B�0R
BIGINT(20)�B���������������interval_young_collection_time (�0 B�0R
BIGINT(20)�9��������������current_thread_count (�0 B�50R
BIGINT(20)���
4����mysql-bin.000008����� ��UTF-80�����-8�B J P��X��������Z��INSERT INTO t_dl_worker_system_state
(worker_id, host, load_average, user_cpu_utilization, sys_cpu_utilization, incoming_network_traffic, outgoing_network_traffic, tcp_current_estab, create_time)
VALUE
(2, '10.104.17.192', 0, 1.26, 0.57, 1313505, 1517376, 101, now())���
c����mysql-bin.000008����� ��UTF-80�����-8�B�datalinkJ�t_dl_worker_system_statePyX�b�
rowsCount��1������G��P b���1� �������������id �(�0 B�978R�BIGINT(20) unsigned�-�������������� worker_id (�0 B�2R
DECIMAL(20,2)�B���������������incoming_network_traffic (�0 B�1313505R
BIGINT(20)�B���������������outgoing_network_traffic (�0 B�1517376R
BIGINT(20)�7���������������tcp_current_estab (�0 B�101R
BIGINT(20)�7� �]��create_time (�0 B�2020-01-17 10:14:59R TIMESTAMP`
如上,会包含正常的数据结果,但是有多于的信息打入kafka
worker的日志如下:
2020-01-17 10:03:24.146 [Task-1-Writer-writer-kafka-chunk-pool-1] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'value.serializer' was supplied but isn't a known config. 2020-01-17 10:03:24.146 [Task-1-Writer-writer-kafka-chunk-pool-1] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'acks' was supplied but isn't a known config. 2020-01-17 10:03:24.146 [Task-1-Writer-writer-kafka-chunk-pool-1] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'batch.size' was supplied but isn't a known config. 2020-01-17 10:03:24.146 [Task-1-Writer-writer-kafka-chunk-pool-1] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'key.serializer' was supplied but isn't a known config
Hi, in DataLink/dl-flinker/dl-flinker-hbasewriter98, there are mulptiple versions of library com.google.protobuf:protobuf-java. However, according to Maven's dependency management strategy: "first declaration wins", only com.google.protobuf:protobuf-java:2.5.0 can be loaded, and com.google.protobuf:protobuf-java:2.6.0 will be shadowed.
In total, there are 10 conflicting API pairs between these two library version.
As shown in the following figure, your project expects to invoke method <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in library com.google.protobuf:protobuf-java:2.6.0 (along the original dependency path). As it has been shadowed, this method defined in com.google.protobuf:protobuf-java:2.5.0 is actually forced to be referenced via the following invocation path (along the actual dependency path):
<com.ucar.datalink.flinker.plugin.writer.hbasewriter98.HbaseUtil: parseColumns(Ljava/util/List;)Ljava/util/List;> /home/wwww/wangSensor/unzip/DataLink-1.0.2-beta/dl-flinker/dl-flinker-hbasewriter98/target/classes
<com.mysql.cj.xdevapi.AbstractDataResult: next()Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.xdevapi.DbDocFactory: createFromProtocolEntity(Lcom/mysql/cj/protocol/ProtocolEntity;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.xdevapi.DbDocFactory: createFromProtocolEntity(Lcom/mysql/cj/protocol/ProtocolEntity;)Lcom/mysql/cj/xdevapi/DbDoc;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.protocol.x.XProtocolRow: getValue(ILcom/mysql/cj/result/ValueFactory;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.protocol.x.XProtocolDecoder: decodeSet([BIILcom/mysql/cj/result/ValueFactory;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.google.protobuf.CodedInputStream: readRawBytes(I)[B>
Although both of these conflicting libraries contain the referenced methods (with the same signature), they have different implementations. This issue will not cause runtime crashes, but it can introduce inconsistent semantic program hehaviors----
Code snippet of <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in com.google.protobuf:protobuf-java:2.6.0 (shadowed but expected to invoke method):
public byte[] readRawBytes(final int size) throws IOException {
final int pos = bufferPos;
if (size <= (bufferSize - pos) && size > 0) {
bufferPos = pos + size;
return Arrays.copyOfRange(buffer, pos, pos + size);
} else {
return readRawBytesSlowPath(size);
}
}
Code snippet of <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in com.google.protobuf:protobuf-java:2.5.0 (loaded version):
public byte[] readRawBytes(final int size) throws IOException {
if (size < 0) {
throw InvalidProtocolBufferException.negativeSize();
}
if (totalBytesRetired + bufferPos + size > currentLimit) {
// Read to the end of the stream anyway.
skipRawBytes(currentLimit - totalBytesRetired - bufferPos);
// Then fail.
throw InvalidProtocolBufferException.truncatedMessage();
}
if (size <= bufferSize - bufferPos) {
// We have all the bytes we need already.
final byte[] bytes = new byte[size];
System.arraycopy(buffer, bufferPos, bytes, 0, size);
bufferPos += size;
return bytes;
} else if (size < BUFFER_SIZE) {
// Reading more bytes than are in the buffer, but not an excessive number
// of bytes. We can safely allocate the resulting array ahead of time.
// First copy what we have.
final byte[] bytes = new byte[size];
int pos = bufferSize - bufferPos;
System.arraycopy(buffer, bufferPos, bytes, 0, pos);
bufferPos = bufferSize;
// We want to use refillBuffer() and then copy from the buffer into our
// byte array rather than reading directly into our byte array because
// the input may be unbuffered.
refillBuffer(true);
while (size - pos > bufferSize) {
System.arraycopy(buffer, 0, bytes, pos, bufferSize);
pos += bufferSize;
bufferPos = bufferSize;
refillBuffer(true);
}
System.arraycopy(buffer, 0, bytes, pos, size - pos);
bufferPos = size - pos;
return bytes;
} else {
// The size is very large. For security reasons, we can't allocate the
// entire byte array yet. The size comes directly from the input, so a
// maliciously-crafted message could provide a bogus very large size in
// order to trick the app into allocating a lot of memory. We avoid this
// by allocating and reading only a small chunk at a time, so that the
// malicious message must actually *be* extremely large to cause
// problems. Meanwhile, we limit the allowed size of a message elsewhere.
// Remember the buffer markers since we'll have to copy the bytes out of
// it later.
final int originalBufferPos = bufferPos;
final int originalBufferSize = bufferSize;
// Mark the current buffer consumed.
totalBytesRetired += bufferSize;
bufferPos = 0;
bufferSize = 0;
// Read all the rest of the bytes we need.
int sizeLeft = size - (originalBufferSize - originalBufferPos);
final List<byte[]> chunks = new ArrayList<byte[]>();
while (sizeLeft > 0) {
final byte[] chunk = new byte[Math.min(sizeLeft, BUFFER_SIZE)];
int pos = 0;
while (pos < chunk.length) {
final int n = (input == null) ? -1 :
input.read(chunk, pos, chunk.length - pos);
if (n == -1) {
throw InvalidProtocolBufferException.truncatedMessage();
}
totalBytesRetired += n;
pos += n;
}
sizeLeft -= chunk.length;
chunks.add(chunk);
}
// OK, got everything. Now concatenate it all into one buffer.
final byte[] bytes = new byte[size];
// Start by copying the leftover bytes from this.buffer.
int pos = originalBufferSize - originalBufferPos;
System.arraycopy(buffer, originalBufferPos, bytes, 0, pos);
// And now all the chunks.
for (final byte[] chunk : chunks) {
System.arraycopy(chunk, 0, bytes, pos, chunk.length);
pos += chunk.length;
}
// Done.
return bytes;
}
}
The detailed informantion of the remaining 9 conflicting API pairs can be found in the following attachment.
10 conflicting API pairs in project dl-flinker-hbasewriter98.txt
[INFO] com.ucar.datalink:dl-flinker-hbasewriter98:jar:1.0.2-beta
[INFO] +- com.ucar.datalink:dl-flinker-api:jar:1.0.2-beta:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] | +- com.google.guava:guava:jar:18.0:compile
[INFO] | +- com.alibaba:fastjson:jar:1.2.62:compile (version managed from 1.1.43)
[INFO] | +- joda-time:joda-time:jar:2.9.4:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.12:compile - version managed from 1.7.7; omitted for duplicate)
[INFO] | +- (ch.qos.logback:logback-classic:jar:1.1.3:compile - version managed from 1.0.13; omitted for duplicate)
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] | +- com.google.code.gson:gson:jar:2.8.0:compile
[INFO] | +- (org.apache.zookeeper:zookeeper:jar:3.4.14:compile - version managed from 3.4.6; omitted for conflict with 3.3.2)
[INFO] | +- com.github.sgroschupf:zkclient:jar:0.1:compile
[INFO] | | +- (org.apache.zookeeper:zookeeper:jar:3.4.14:compile - version managed from 3.3.3; omitted for duplicate)
[INFO] | | - log4j:log4j:jar:1.2.14:compile
[INFO] | +- org.springframework:spring:jar:2.5.6:compile
[INFO] | | - commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | - mysql:mysql-connector-java:jar:8.0.11:compile
[INFO] | \ - (com.google.protobuf:protobuf-java:jar:2.6.0:compile - scope updated from runtime; omitted for duplicate)
[INFO] +- org.slf4j:slf4j-api:jar:1.7.12:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.1.3:compile
[INFO] | +- ch.qos.logback:logback-core:jar:1.1.3:compile
[INFO] | - (org.slf4j:slf4j-api:jar:1.7.12:compile - version managed from 1.7.7; omitted for duplicate)
[INFO] +- org.apache.hbase:hbase-client:jar:0.98.16.1-hadoop1:compile
[INFO] | +- org.apache.hbase:hbase-annotations:jar:0.98.16.1-hadoop1:compile
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile - omitted for duplicate)
[INFO] | | - (junit:junit:jar:4.12:test - version managed from 4.11; scope managed from compile; omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-common:jar:0.98.16.1-hadoop1:compile
[INFO] | | +- (org.apache.hbase:hbase-annotations:jar:0.98.16.1-hadoop1:compile - omitted for duplicate)
......
[INFO] | +- commons-codec:commons-codec:jar:1.7:compile
[INFO] | +- (commons-io:commons-io:jar:2.4:compile - version managed from 2.1; omitted for duplicate)
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- (com.google.guava:guava:jar:18.0:compile - version managed from 12.0.1; omitted for duplicate)
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | +- io.netty:netty:jar:3.10.6.Final:compile
......
Solution: Declare version com.google.protobuf:protobuf-java:jar:2.6.0 as a direct dependency, to override the version 2.5.0 (based on Maven's nearest wins loading strategy).
Thanks.
Best regards,
Coco
介质管理 -> RDBMS -> 新增 Etl-Host 不填写时, 会创建失败, 检查代码, 发现是要获取此配置项的值连接数据库 。 如果此为必选项建议强制用户填写。 若非必选项, 建议代码逻辑去除此配置项的连接校验。
增量任务-> Task 管理 -> MysqlTask -> Taks 名称 为例, 当用户填入任意名称时,提示必须以xxx 为前缀, 建议可以写入默认值前缀引导用户填写或鼠标接近输入框时自动提示规则。
启动worker 时, 提示worker/plugin/reader , worker/plugin/writer 等目录不存在, 手动创建目录后, 又提示对应的插件jar 包不存在。 建议可以考虑优化下。 例如在工程打包时预置一些开箱即用的包。
com.ucar.datalink.common.errors.DataLinkException:Worker is not found for client id [null] or ip [172.21.0.1]
环境:
centos7
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
目前的解决办法是曲线救国:
在work conf设置client.id=1 (此1为Manager的web管理界面的菜单:集群管理->机器管理->机器ID)
执行全量任务,postgresql to mysql时候出现如下报错,请问下应该怎样解决?
dl-flinker组件下的admin.log日志里出现
DataxAdminLauncher start job processor failure,please check job config file or start script file !
process failed for job ad_day_data_N6jpZhNMTt.Error Details:
Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is
com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
#版本 V1.0.1-beta
向源数据库表插入数据完毕
查看目标表数据没有同步数据。
Manager Log:
2020-01-16 18:05:24.131 [qtp1792384402-147] INFO c.u.d.m.core.web.controller.shadow.TaskShadowController - 收到的参数为:{search={"regex":false,"value":""}, columns=[{"search":{"regex":false,"value":""},"data":"id","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"taskName","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"state","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"mappingIds","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"resetTime","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"createTime","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"modifyTime","orderable":true,"name":"","searchable":true}], start=0, length=10, draw=1, state=, taskId=4, order=[{"column":0,"dir":"asc"}]}
Worker Log:
java.lang.NullPointerException: null
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl.getWorkerSystemState(WorkerSystemStateProbeImpl.java:95) ~[dl-worker-core-1.0.0-beta.jar:na]
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl.access$000(WorkerSystemStateProbeImpl.java:31) ~[dl-worker-core-1.0.0-beta.jar:na]
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl$1.run(WorkerSystemStateProbeImpl.java:65) ~[dl-worker-core-1.0.0-beta.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
2020-01-16 18:13:59.130 [Probe-Type-WorkerSystemStateProbe-pool-1] ERROR c.u.d.worker.core.probe.WorkerSystemStateProbeImpl - Worker system state probe failed.
java.lang.NullPointerException: null
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl.getWorkerSystemState(WorkerSystemStateProbeImpl.java:95) ~[dl-worker-core-1.0.0-beta.jar:na]
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl.access$000(WorkerSystemStateProbeImpl.java:31) ~[dl-worker-core-1.0.0-beta.jar:na]
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl$1.run(WorkerSystemStateProbeImpl.java:65) ~[dl-worker-core-1.0.0-beta.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Task Log:
2020-01-16 17:58:07.404 [Task-5-Reader-reader-mysql] INFO com.ucar.datalink.worker.core.runtime.WorkerTaskReader - TaskReader-reader-mysql finished initialization and start.
2020-01-16 17:58:07.442 [destination = 5-main , address = /172.16.159.139:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> begin to find start position, it will be long time for reset or first position
2020-01-16 17:58:07.443 [destination = 5-main , address = /172.16.159.139:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position just show master status
2020-01-16 17:58:07.444 [destination = 5-main , address = /172.16.159.139:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000013,position=945,serverId=,gtid=,timestamp=] cost : 1ms , the next step is binlog dump
能通过idea启动main方法,打开界面
但是打包之后,启动报错
2020-02-26 14:25:52.726 [main] WARN o.s.web.context.support.XmlWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mediaMappingChangeEventHandler' defined in URL [jar:file:/home/zs/datalink/dl-manager/lib/dl-manager-core-1.0.0-beta.jar!/com/ucar/datalink/manager/core/handler/MediaMappingChangeEventHandler.class]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1101) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:663) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:629) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:677) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:548) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:489) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:136) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at javax.servlet.GenericServlet.init(GenericServlet.java:244) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:616) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:396) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:871) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) [jetty-webapp-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) [jetty-webapp-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) [jetty-webapp-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at com.ucar.datalink.manager.core.server.JettyServer.startup(JettyServer.java:57) [dl-manager-core-1.0.0-beta.jar:na]
at com.ucar.datalink.manager.core.server.ServerContainer.startup(ServerContainer.java:65) [dl-manager-core-1.0.0-beta.jar:na]
at com.ucar.datalink.manager.core.boot.ManagerBootStrap.boot(ManagerBootStrap.java:38) [dl-manager-core-1.0.0-beta.jar:na]
at com.ucar.datalink.manager.core.boot.ManagerBootStrap.main(ManagerBootStrap.java:28) [dl-manager-core-1.0.0-beta.jar:na]
另外对于数据库密码明文,在idea里面直接使用明文是可行的,打包之后会报密码长度问题
line 2: $'\r': command not found
遇到了数据无法写入的问题。请求帮忙,万分感谢!
按照Idea QuickStart一步步启动了Manager和Worker,Worker按照Distributed模式启动
源数据库和目的地数据库均为mysql,且为同一个实例不同的Database
配置了2个介质(数据源),同一个mysql实例不同的数据库,且读库写库配置的完全一致
机器管理显示机器状态正常,后台输出正常
任务列表如下,状态显示正常
任务ID | 任务名称 | 详情 | 目标状态 | 实际状态 | 所属分组 | 所属机器 | 当前日志时间 | 最近启动时间 | 操作 |
---|---|---|---|---|---|---|---|---|---|
2 | test_base_2_1 | Details | STARTED | RUNNING | 1 | 1 | 2019-09-24 16:03:32 (1569312212000) | 2019-09-24 16:03:33 |
映射配置使用白名单,字段完全对应。
2019-04-25 15:16:23.643 [main] ERROR com.ucar.datalink.common.utils.DbConfigEncryption - something goes wrong when decrypt
java.lang.NumberFormatException: For input string: "pa"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.8.0_121]
at java.lang.Integer.parseInt(Integer.java:580) ~[na:1.8.0_121]
at com.ucar.datalink.common.utils.DbConfigEncryption.hexStr2ByteArr(DbConfigEncryption.java:104) ~[dl-common-0.0.2.jar:na]
at com.ucar.datalink.common.utils.DbConfigEncryption.decrypt(DbConfigEncryption.java:45) ~[dl-common-0.0.2.jar:na]
at com.ucar.datalink.biz.utils.DLPropertiesPersister.load(DLPropertiesPersister.java:24) [dl-biz-0.0.2.jar:na]
at org.springframework.core.io.support.PropertiesLoaderUtils.fillProperties(PropertiesLoaderUtils.java:99) [spring-core-4.1.6.RELEASE.jar
:4.1.6.RELEASE]
请问支持mysql5.7.22版本?我想试用一下
经测试写RDBMS数据库postgreSQL只能写入public模式下
模块:manager-core
jetty:9.4.15
配置好jetty容器,发布manager-core
出现以下异常,涉及类加载流程中,tomcat-util包的冲突问题。最终页面503,不可访问
Caused by: java.util.ServiceConfigurationError: org.apache.juli.logging.Log: Provider org.eclipse.jetty.apache.jsp.JuliLog not a subtype at java.util.ServiceLoader.fail(ServiceLoader.java:239) at java.util.ServiceLoader.access$300(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.juli.logging.LogFactory.<init>(LogFactory.java:78)
请问是jetty版本使用过高吗?
2019-07-29 10:25:18.013 [main] WARN o.s.b.factory.config.PreferencesPlaceholderConfigurer - Could not load properties from class path resource [con
f/datasource.properties]: class path resource [conf/datasource.properties] cannot be opened because it does not exist
2019-07-29 10:25:18.013 [main] WARN o.s.b.factory.config.PreferencesPlaceholderConfigurer - Could not load properties from class path resource [con
f/biz.properties]: class path resource [conf/biz.properties] cannot be opened because it does not exist
2019-07-29 10:25:18.013 [main] WARN o.s.b.factory.config.PreferencesPlaceholderConfigurer - Could not load properties from class path resource [con
f/extend.properties]: class path resource [conf/extend.properties] cannot be opened because it does not exist
这个警告有影响吗?
还有一个问题是:
2019-04-25 15:16:23.643 [main] ERROR com.ucar.datalink.common.utils.DbConfigEncryption - something goes wrong when decrypt
在配置中我使用的明文密码,应该是不需要解密的,这个报错应该不影响吗?
最后一个问题是“worker启动的时候,client.id,要么从本地配置文件获取,要么从数据库通过ip反向查询”,但是:我在manger端配置的时候用的公网IP,而worker启动的时候IP反向查询时自动用的是局域网IP(192开头的),导致数据库中反向查询不到,这个有什么办法解决?
[main] ERROR com.ucar.datalink.worker.core.boot.WorkerBootStrap - ## Something goes wrong when starting up the datalink work
er:
com.ucar.datalink.common.errors.DatalinkException: Worker is not found for client id [null] or ip [192...*]
t_dl_flinker_job_run_queue表的修改时间(modify_time)默认值不同统一,应为
modify_time : CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT
create_time : CURRENT_TIMESTAMP COMMENT
初始化DB时使用HS导入,报错。
端口号应该新增 应该分开读库和写库的让用户自定义端口 要么直接和host写在一起比如:localhost:3306
或者localhost:4406
您好,请问一下:在1.8_深入数据一致性中有提到“数据最终会恢复到最后一个事件对应的状态”,那么是否意味着数据可能会存在短暂的回退现象?比如原本 x=1, x=2, x=3 的操作,最终反映到目的端的时候会出现 x=3, x=1, x=2, x=3 这样的变化现象?Thanks
大数据组件web管理系统太多了,自建用户成本比较高,希望能支持一些标准用户登录体系?
如标题所述
#版本 v1.1.0-beta
为啥我worker启动成功了,执行任务前端报错no found zk workers ip
startup.sh的脚本里面都配置了远程的jmx端口:Dcom.sun.management.jmxremote.port=9933
work和manger部署在一起会冲突,或者多work的形式也会冲突应该做成多个变量的形式让用户传入。
我在windows 10上运行,进入power shell,执行.\startup.sh,出现CMD界面一瞬即逝。看日志manager.log中只有如下描述:
bash: /c/Program: No such file or directory
bash: /c/Program: No such file or directory
重启任务时id获取到的值是错误的,导致重启报错
2019-12-11 17:56:14.777 [main] ERROR com.ucar.datalink.worker.core.boot.WorkerBootStrap - ## Something goes wrong when starting up the datalink worker:
com.ucar.datalink.common.errors.DatalinkException: Worker is not found for client id [null] or ip [10.11.0.176]
at com.ucar.datalink.worker.core.boot.WorkerBootStrap.buildWorkerProps(WorkerBootStrap.java:151) [dl-worker-core-0.0.2.jar:na]
at com.ucar.datalink.worker.core.boot.WorkerBootStrap.boot(WorkerBootStrap.java:52) [dl-worker-core-0.0.2.jar:na]
at com.ucar.datalink.worker.core.boot.WorkerBootStrap.main(WorkerBootStrap.java:198) [dl-worker-core-0.0.2.jar:na]
1、admin后台的初始化sql有两处异常;【bug】
2、json的问题(包括后台和worker都会出现),https://github.com/alibaba/fastjson/wiki/enable_autotype,我暂时处理办法是变更了部分代码做降级处理【bug】
3、后台task查询时候join表的时候会出现别名重复问题,mapper-task.xml中的listByCondition【bug】
4、mysql大小写敏感问题,建议统一换成小写【建议】
我提个建议:我认为默认就是填写明文,程序启动后读取到内存中,然后再改写配置文件成为密文即可,还可以在这基础上加一个配置 是否加密。
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.