ucargroup/datalink

DataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步，分布式、可扩展的数据交换平台。

项目介绍

DataLink是一个满足各种异构数据源之间的实时增量同步、离线全量同步，分布式、可扩展的数据交换平台。

项目背景

着眼于未来，我们的目标是打造一个平台，满足各种异构数据源之间的实时增量同步和离线全量同步，支撑公司业务的快速发展。在充分调研的基础之上，我们发现，没有任何一款开源产品能轻易的满足我们的目标，每个产品都有其明显的短板和局限性，所以最终的选项只有"自行设计"。但自行设计并不是凭空设计，现有的数据交换平台、已有的经验、大大小小的开源产品都是我们的设计根基，与其说是自行设计，倒不如说是站在巨人的肩膀上做了一次飞跃。由此诞生了DataLink这样一个产品：

• 满足各种异构数据源之间的实时增量同步和离线全量同步
• 平台提供统一的基础设施（高可用、动态负载、同步任务管理、插件管理、监控报警、公用业务组件等等），让设计人员专注于同步插件开发，一次投入，长久受益
• 吸收、整合业内经验，在架构模型、设计方法论、功能特性、可运维、易用性上进行全面的升级，在前瞻性和扩展性上下足功夫，满足未来5-10年内的各种同步需求

DataLink开发时间从2016年12月开始，第一版于2017年5月份上线，在神州优车集团服役到现在，基本上满足了公司所有业务线的同步需求。此次外部开源版本为去除内部依赖后的版本。

内部使用情况

• 支持了神州优车和瑞幸咖啡的核心业务运行
• 涉及500多个数据库实例之间的6000+个同步映射
• 100台Worker+2台Manager机器的集群规模
• 日均数据同步量TB级

当前规划

整合业内更多的产品经验，打造一个全新的平台，功能丰富程度、扩展性、标准化层面更上一层楼

相关文章

神州优车数据交换平台的架构、建设与痛点难点详解
https://mp.weixin.qq.com/s/BVuDbS-2Ra5pIJ7oV78FBA
（神州优车）大数据平台建设经验分享
https://www.cnblogs.com/ucarinc/p/12091053.html

总体架构

工作原理(增量)

• 典型Master-Slave式系统架构，Manager(Web管理)+Worker(工作节点)
  a. Manager负责worker的负载均衡、集群的配置管理和系统监控
  b. Worker核心功能是管理Task的生命周期，并配合Manager进行Re-Balance
  
• Zookeeper：Manager的高可用需要依赖于Zookeeper，另外，Task会将运行时信息注册到Zookeeper
• Mysql：Datalink的运行需要依赖各种配置信息、以及在运行过程中会动态产生监控和统计数据，统一保存到Mysql中

QuickStart

See the page for quick start: QuickStart

架构&文档

See the page for introduction: 架构&文档

常见问题

See the page for FAQ: FAQ

类似开源

canal：http://github.com/alibaba/canal
otter：https://github.com/alibaba/otter
Kafka-Connect：https://github.com/apache/kafka
DataBus：https://github.com/linkedin/databus

版本历史

当前最新版本为1.0.2-beta，版本发布历史如下：
https://github.com/ucarGroup/DataLink/releases/

问题反馈

目前有关DataLink的问题交流方式有如下几种，欢迎各位加入进行技术讨论。
qq交流群： 758937055
邮件交流： [email protected]
报告issue：issues

datalink's Issues

toRestartMysqlTask 报错

重启任务时id获取到的值是错误的，导致重启报错

Task 运行时异常退出

Task 运行时异常退出，如上图。 暂不知原因。

介质管理RDBMS 新增任务报错

#版本 v1.1.0-beta

1. 介质管理新增RDBMS 任务报错。
   #1 确认源数据库用户名和密码正确。
   后台日志信息截取：
   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
   at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:172) ~[mysql-connector-java-8.0.11.jar:8.0.11]
   at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64) ~[mysql-connector-java-8.0.11.jar:8.0.11]
   at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:862) ~[mysql-connector-java-8.0.11.jar:8.0.11]
   at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:444) ~[mysql-connector-java-8.0.11.jar:8.0.11]
   at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:230) ~[mysql-connector-java-8.0.11.jar:8.0.11]
   at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:226) ~[mysql-connector-java-8.0.11.jar:8.0.11]
   at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[na:1.8.0_121]
   at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[na:1.8.0_121]
   at com.ucar.datalink.biz.meta.RDBMSUtil.checkConnection(RDBMSUtil.java:558) ~[dl-biz-1.0.0-beta.jar:na]
   at com.ucar.datalink.biz.meta.RDBMSUtil.checkRdbConnection(RDBMSUtil.java:396) ~[dl-biz-1.0.0-beta.jar:na]
   at com.ucar.datalink.biz.service.impl.MediaSourceServiceImpl.checkDbConnection(MediaSourceServiceImpl.java:218) ~[dl-biz-1.0.0-beta.jar:na]
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
   at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) ~[spring-aop-4.1.6.RELEASE.jar:4.1.6.RELEASE]
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201) ~[spring-aop-4.1.6.RELEASE.jar:4.1.6.RELEASE]
   at com.sun.proxy.$Proxy34.checkDbConnection(Unknown Source) ~[na:na]
   at com.ucar.datalink.manager.core.web.controller.mediaSource.MediaSourceController.doAdd(MediaSourceController.java:158) ~[dl-manager-core-1.0.0-beta.jar:na]

打包后worker目录与压缩包内容不一致

写RDBMS

经测试写RDBMS数据库postgreSQL只能写入public模式下

请问关于oracle-reader的计划?

现在的项目是否不支持从oracle中读取数据?
然后是否有计划支持?
如果可以共建的话,怎样加入?
个人想学习交流

启动work报错

2019-04-25 15:16:23.643 [main] ERROR com.ucar.datalink.common.utils.DbConfigEncryption - something goes wrong when decrypt
java.lang.NumberFormatException: For input string: "pa"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[na:1.8.0_121]
at java.lang.Integer.parseInt(Integer.java:580) ~[na:1.8.0_121]
at com.ucar.datalink.common.utils.DbConfigEncryption.hexStr2ByteArr(DbConfigEncryption.java:104) ~[dl-common-0.0.2.jar:na]
at com.ucar.datalink.common.utils.DbConfigEncryption.decrypt(DbConfigEncryption.java:45) ~[dl-common-0.0.2.jar:na]
at com.ucar.datalink.biz.utils.DLPropertiesPersister.load(DLPropertiesPersister.java:24) [dl-biz-0.0.2.jar:na]
at org.springframework.core.io.support.PropertiesLoaderUtils.fillProperties(PropertiesLoaderUtils.java:99) [spring-core-4.1.6.RELEASE.jar
:4.1.6.RELEASE]

用户管理问题

新版本用户管理报错，0.2版本智能查看admin用户，无法新增或者修改用户信息

启动manger或者worker都会警告properties路径不存在

2019-07-29 10:25:18.013 [main] WARN o.s.b.factory.config.PreferencesPlaceholderConfigurer - Could not load properties from class path resource [con
f/datasource.properties]: class path resource [conf/datasource.properties] cannot be opened because it does not exist
2019-07-29 10:25:18.013 [main] WARN o.s.b.factory.config.PreferencesPlaceholderConfigurer - Could not load properties from class path resource [con
f/biz.properties]: class path resource [conf/biz.properties] cannot be opened because it does not exist
2019-07-29 10:25:18.013 [main] WARN o.s.b.factory.config.PreferencesPlaceholderConfigurer - Could not load properties from class path resource [con
f/extend.properties]: class path resource [conf/extend.properties] cannot be opened because it does not exist

这个警告有影响吗？

还有一个问题是：
2019-04-25 15:16:23.643 [main] ERROR com.ucar.datalink.common.utils.DbConfigEncryption - something goes wrong when decrypt
在配置中我使用的明文密码，应该是不需要解密的，这个报错应该不影响吗？

最后一个问题是“worker启动的时候，client.id，要么从本地配置文件获取，要么从数据库通过ip反向查询”，但是：我在manger端配置的时候用的公网IP，而worker启动的时候IP反向查询时自动用的是局域网IP（192开头的），导致数据库中反向查询不到，这个有什么办法解决？
[main] ERROR com.ucar.datalink.worker.core.boot.WorkerBootStrap - ## Something goes wrong when starting up the datalink work
er:
com.ucar.datalink.common.errors.DatalinkException: Worker is not found for client id [null] or ip [192...*]

worker运行bug以及kafka消息乱码

reader是mysql，write是kafka，运行中报如下错误以及消费kafka是乱码

1、运行时候报错，出现如下错误（已解决）
Caused by: java.lang.LinkageError: loader constraint violation: when resolving method "org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;" the class loader (instance of com/ucar/datalink/worker/core/util/classloader/RelPluginClassLoader) of the current class, org/slf4j/LoggerFactory, and the class loader (instance of sun/misc/Launcher$AppClassLoader) for the method's defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature

解决办法：kafka的plugin的lib依赖去除sl4j依赖包即可

2、kafka消息为乱码数据，如下片段
{"databaseName":"test","dbTableRowCellVOList":[{"afterValue":"啊啊啊啊啊啊","beforeValue":"中文啊啊啊","columnName":"name"},{"afterValue":"1","columnName":"id"}],"eventType":"UPDATE","id":"1","tableName":"test1"} ��*����� �| 3����mysql-bin.000008���� �*�UTF-80�ڠ��-8�B J PVX����C��Z?UPDATE test1SETname='啊啊啊啊啊啊' WHERE (id='1')��� L����mysql-bin.000008����� �*�UTF-80�ڠ��-8�B�testJ�test1PYX�b� rowsCount��1������I��P b�� &� �������������id �( 0 B�1R bigint(20) /�����name ( 0 B�中文啊啊啊Rvarchar(100)�&� �������������id �( 0 B�1R bigint(20)�2�����name (�0 B�啊啊啊啊啊啊Rvarchar(100)�@ 1����mysql-bin.000008�Е�� �*�UTF-80�ڠ��-8�B J P���� ��8104191 ��*����� ��� 4����mysql-bin.000008����� �*�UTF-80�����-8�B J P��X����r��ZnINSERT INTO t_dl_task_delaytime (task_id,delay_time,create_time) VALUES (1,633, now())��� ^����mysql-bin.000008���� �*�UTF-80�����-8�B�datalinkJ�t_dl_task_delaytimeP?X�b� rowsCount��1������7��P b���/� �������������id �(�0 B�6R�BIGINT(20) unsigned�0�������������� delay_time (�0 B�633R BIGINT(20)�6���]��create_time (�0 B�2020-01-17 10:14:59R�DATETIME�+���������������task_id (�0 B�1R BIGINT(20)��� 4����mysql-bin.000008���� �*�UTF-80�����-8�B J P��X��������Z��INSERT INTO t_dl_task_statistic (task_id, records_per_minute, size_per_minute, write_time_per_record, exceptions_per_minute, read_write_count_per_minute, create_time) VALUES (1, 2, 398, 4.00, 0, 2, now())��� ^����mysql-bin.000008����� �*�UTF-80�����-8�B�datalinkJ�t_dl_task_statisticPX�b�
rowsCount��1������;��P b���1� �������������id �(�0 B�970R�BIGINT(20) unsigned�+���������������task_id (�0 B�1R
BIGINT(20)�6���������������records_per_minute (�0 B�2R
BIGINT(20)�5���������������size_per_minute (�0 B�398R
DECIMAL(20,2)�9���������������exceptions_per_minute (�0 B�0R
BIGINT(20)�6���]��create_time (�0 B�2020-01-17 10:14:59R�DATETIME�?���������������read_write_count_per_minute (�0 B�2R
BIGINT(20)���
4����mysql-bin.000008�ܜ�� ��UTF-80�����-8�B J P��X��������Z��INSERT INTO t_dl_worker_jvm_state
(worker_id, host, old_mem_used, old_mem_max, young_mem_max, young_mem_used, create_time, interval_old_collection_count, interval_young_collection_count, interval_old_collection_time, interval_young_collection_time, current_thread_count)
VALUE
(2, '10.104.17.192', 0, 4294967296, 1610612736, 611795072, now(), 0, 0, 0, 0, 50)���
a����mysql-bin.000008�Š�� ��UTF-80�����-8�B�datalinkJ�t_dl_worker_jvm_stateP��X�b�
rowsCount��1������F��P b���1� �������������id �(�0 B�978R�BIGINT(20) unsigned�-�������������� worker_id (�0 B�2R
10.104.17.192R�VARCHAR(20)�0��������������old_mem_used (�0 B�0R
BIGINT(20)�8���������������old_mem_max (�0 B
4294967296R
young_mem_max (�0 B
1610612736R
BIGINT(20)�:���������������young_mem_used (�0 B 611795072R
BIGINT(20)�7���]��create_time (�0 B�2020-01-17 10:14:59R TIMESTAMP�A���������������interval_old_collection_count (�0 B�0R
BIGINT(20)�C� �������������interval_young_collection_count (�0 B�0R
BIGINT(20)�@�
�������������interval_old_collection_time (�0 B�0R
BIGINT(20)�B���������������interval_young_collection_time (�0 B�0R
BIGINT(20)�9��������������current_thread_count (�0 B�50R
BIGINT(20)���
4����mysql-bin.000008����� ��UTF-80�����-8�B J P��X��������Z��INSERT INTO t_dl_worker_system_state
(worker_id, host, load_average, user_cpu_utilization, sys_cpu_utilization, incoming_network_traffic, outgoing_network_traffic, tcp_current_estab, create_time)
VALUE
(2, '10.104.17.192', 0, 1.26, 0.57, 1313505, 1517376, 101, now())���
c����mysql-bin.000008����� ��UTF-80�����-8�B�datalinkJ�t_dl_worker_system_statePyX�b�
rowsCount��1������G��P b���1� �������������id �(�0 B�978R�BIGINT(20) unsigned�-�������������� worker_id (�0 B�2R
DECIMAL(20,2)�B���������������incoming_network_traffic (�0 B�1313505R
BIGINT(20)�B���������������outgoing_network_traffic (�0 B�1517376R
BIGINT(20)�7���������������tcp_current_estab (�0 B�101R
BIGINT(20)�7� �]��create_time (�0 B�2020-01-17 10:14:59R TIMESTAMP`

如上，会包含正常的数据结果，但是有多于的信息打入kafka

worker的日志如下：
2020-01-17 10:03:24.146 [Task-1-Writer-writer-kafka-chunk-pool-1] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'value.serializer' was supplied but isn't a known config. 2020-01-17 10:03:24.146 [Task-1-Writer-writer-kafka-chunk-pool-1] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'acks' was supplied but isn't a known config. 2020-01-17 10:03:24.146 [Task-1-Writer-writer-kafka-chunk-pool-1] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'batch.size' was supplied but isn't a known config. 2020-01-17 10:03:24.146 [Task-1-Writer-writer-kafka-chunk-pool-1] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'key.serializer' was supplied but isn't a known config

提一个我认为是 Bug，顺带给出解决方案（有可能作者不一定认为）

遇到问题场景

• 在 CentOS 7.4 执行 startup.sh，发现报：line 2: $'\r': command not found
• 很明显，在 sh 文件上采用 CRLF

继续排查

• Clone 了代码，查看了所有 sh 后缀的脚本
• IntelliJ IDEA 显示所有 sh 后缀的脚本都是 CRLF

个人想法

• 我个人对 Linux 的理解是：所有 sh 脚本应该都是 LF 的，但是不知道作者是怎样考虑的。

解决办法

• 如果作者用 IntelliJ IDEA 开发，如下图操作即可，然后提交代码。

感谢

• 最近有在看该工具，作者用心了，至少代码和文档看着不会让人感觉很变扭
• IntelliJ IDEA 打开，轻松 build、install 没有任何报错

脚本远程的问题

startup.sh的脚本里面都配置了远程的jmx端口：Dcom.sun.management.jmxremote.port=9933

work和manger部署在一起会冲突，或者多work的形式也会冲突应该做成多个变量的形式让用户传入。

针对V1.0.0-beta版本部分问题记录

1、admin后台的初始化sql有两处异常；【bug】
2、json的问题（包括后台和worker都会出现），https://github.com/alibaba/fastjson/wiki/enable_autotype，我暂时处理办法是变更了部分代码做降级处理【bug】
3、后台task查询时候join表的时候会出现别名重复问题，mapper-task.xml中的listByCondition【bug】
4、mysql大小写敏感问题，建议统一换成小写【建议】

Dependency conflicts on com.google.protobuf:protobuf-java, leading to inconsistent program behaviors

Hi, in DataLink/dl-flinker/dl-flinker-hbasewriter98, there are mulptiple versions of library com.google.protobuf:protobuf-java. However, according to Maven's dependency management strategy: "first declaration wins", only com.google.protobuf:protobuf-java:2.5.0 can be loaded, and com.google.protobuf:protobuf-java:2.6.0 will be shadowed.

In total, there are 10 conflicting API pairs between these two library version.

As shown in the following figure, your project expects to invoke method <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in library com.google.protobuf:protobuf-java:2.6.0 (along the original dependency path). As it has been shadowed, this method defined in com.google.protobuf:protobuf-java:2.5.0 is actually forced to be referenced via the following invocation path (along the actual dependency path):

<com.ucar.datalink.flinker.plugin.writer.hbasewriter98.HbaseUtil: parseColumns(Ljava/util/List;)Ljava/util/List;> /home/wwww/wangSensor/unzip/DataLink-1.0.2-beta/dl-flinker/dl-flinker-hbasewriter98/target/classes
<com.mysql.cj.xdevapi.AbstractDataResult: next()Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.xdevapi.DbDocFactory: createFromProtocolEntity(Lcom/mysql/cj/protocol/ProtocolEntity;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.xdevapi.DbDocFactory: createFromProtocolEntity(Lcom/mysql/cj/protocol/ProtocolEntity;)Lcom/mysql/cj/xdevapi/DbDoc;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.protocol.x.XProtocolRow: getValue(ILcom/mysql/cj/result/ValueFactory;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.mysql.cj.protocol.x.XProtocolDecoder: decodeSet([BIILcom/mysql/cj/result/ValueFactory;)Ljava/lang/Object;> /home/wwww/.m2/repository/mysql/mysql-connector-java/8.0.11/mysql-connector-java-8.0.11.jar
<com.google.protobuf.CodedInputStream: readRawBytes(I)[B>

Although both of these conflicting libraries contain the referenced methods (with the same signature), they have different implementations. This issue will not cause runtime crashes, but it can introduce inconsistent semantic program hehaviors----

Code snippet of <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in com.google.protobuf:protobuf-java:2.6.0 (shadowed but expected to invoke method):

detailed method body

public byte[] readRawBytes(final int size) throws IOException {
    final int pos = bufferPos;
    if (size <= (bufferSize - pos) && size > 0) {
      bufferPos = pos + size;
      return Arrays.copyOfRange(buffer, pos, pos + size);
    } else {
      return readRawBytesSlowPath(size);
    }
  }

Code snippet of <com.google.protobuf.CodedInputStream: readRawBytes(I)[B> in com.google.protobuf:protobuf-java:2.5.0 (loaded version):

detailed method body

public byte[] readRawBytes(final int size) throws IOException {
    if (size < 0) {
      throw InvalidProtocolBufferException.negativeSize();
    }

    if (totalBytesRetired + bufferPos + size > currentLimit) {
      // Read to the end of the stream anyway.
      skipRawBytes(currentLimit - totalBytesRetired - bufferPos);
      // Then fail.
      throw InvalidProtocolBufferException.truncatedMessage();
    }

    if (size <= bufferSize - bufferPos) {
      // We have all the bytes we need already.
      final byte[] bytes = new byte[size];
      System.arraycopy(buffer, bufferPos, bytes, 0, size);
      bufferPos += size;
      return bytes;
    } else if (size < BUFFER_SIZE) {
      // Reading more bytes than are in the buffer, but not an excessive number
      // of bytes.  We can safely allocate the resulting array ahead of time.

      // First copy what we have.
      final byte[] bytes = new byte[size];
      int pos = bufferSize - bufferPos;
      System.arraycopy(buffer, bufferPos, bytes, 0, pos);
      bufferPos = bufferSize;

      // We want to use refillBuffer() and then copy from the buffer into our
      // byte array rather than reading directly into our byte array because
      // the input may be unbuffered.
      refillBuffer(true);

      while (size - pos > bufferSize) {
        System.arraycopy(buffer, 0, bytes, pos, bufferSize);
        pos += bufferSize;
        bufferPos = bufferSize;
        refillBuffer(true);
      }

      System.arraycopy(buffer, 0, bytes, pos, size - pos);
      bufferPos = size - pos;

      return bytes;
    } else {
      // The size is very large.  For security reasons, we can't allocate the
      // entire byte array yet.  The size comes directly from the input, so a
      // maliciously-crafted message could provide a bogus very large size in
      // order to trick the app into allocating a lot of memory.  We avoid this
      // by allocating and reading only a small chunk at a time, so that the
      // malicious message must actually *be* extremely large to cause
      // problems.  Meanwhile, we limit the allowed size of a message elsewhere.

      // Remember the buffer markers since we'll have to copy the bytes out of
      // it later.
      final int originalBufferPos = bufferPos;
      final int originalBufferSize = bufferSize;

      // Mark the current buffer consumed.
      totalBytesRetired += bufferSize;
      bufferPos = 0;
      bufferSize = 0;

      // Read all the rest of the bytes we need.
      int sizeLeft = size - (originalBufferSize - originalBufferPos);
      final List<byte[]> chunks = new ArrayList<byte[]>();

      while (sizeLeft > 0) {
        final byte[] chunk = new byte[Math.min(sizeLeft, BUFFER_SIZE)];
        int pos = 0;
        while (pos < chunk.length) {
          final int n = (input == null) ? -1 :
            input.read(chunk, pos, chunk.length - pos);
          if (n == -1) {
            throw InvalidProtocolBufferException.truncatedMessage();
          }
          totalBytesRetired += n;
          pos += n;
        }
        sizeLeft -= chunk.length;
        chunks.add(chunk);
      }

      // OK, got everything.  Now concatenate it all into one buffer.
      final byte[] bytes = new byte[size];

      // Start by copying the leftover bytes from this.buffer.
      int pos = originalBufferSize - originalBufferPos;
      System.arraycopy(buffer, originalBufferPos, bytes, 0, pos);

      // And now all the chunks.
      for (final byte[] chunk : chunks) {
        System.arraycopy(chunk, 0, bytes, pos, chunk.length);
        pos += chunk.length;
      }

      // Done.
      return bytes;
    }
  }

The detailed informantion of the remaining 9 conflicting API pairs can be found in the following attachment.
10 conflicting API pairs in project dl-flinker-hbasewriter98.txt

Dependency tree--

[INFO] com.ucar.datalink:dl-flinker-hbasewriter98:jar:1.0.2-beta
[INFO] +- com.ucar.datalink:dl-flinker-api:jar:1.0.2-beta:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] | +- com.google.guava:guava:jar:18.0:compile
[INFO] | +- com.alibaba:fastjson:jar:1.2.62:compile (version managed from 1.1.43)
[INFO] | +- joda-time:joda-time:jar:2.9.4:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.12:compile - version managed from 1.7.7; omitted for duplicate)
[INFO] | +- (ch.qos.logback:logback-classic:jar:1.1.3:compile - version managed from 1.0.13; omitted for duplicate)
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] | +- com.google.code.gson:gson:jar:2.8.0:compile
[INFO] | +- (org.apache.zookeeper:zookeeper:jar:3.4.14:compile - version managed from 3.4.6; omitted for conflict with 3.3.2)
[INFO] | +- com.github.sgroschupf:zkclient:jar:0.1:compile
[INFO] | | +- (org.apache.zookeeper:zookeeper:jar:3.4.14:compile - version managed from 3.3.3; omitted for duplicate)
[INFO] | | - log4j:log4j:jar:1.2.14:compile
[INFO] | +- org.springframework:spring:jar:2.5.6:compile
[INFO] | | - commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | - mysql:mysql-connector-java:jar:8.0.11:compile
[INFO] | \ - (com.google.protobuf:protobuf-java:jar:2.6.0:compile - scope updated from runtime; omitted for duplicate)
[INFO] +- org.slf4j:slf4j-api:jar:1.7.12:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.1.3:compile
[INFO] | +- ch.qos.logback:logback-core:jar:1.1.3:compile
[INFO] | - (org.slf4j:slf4j-api:jar:1.7.12:compile - version managed from 1.7.7; omitted for duplicate)
[INFO] +- org.apache.hbase:hbase-client:jar:0.98.16.1-hadoop1:compile
[INFO] | +- org.apache.hbase:hbase-annotations:jar:0.98.16.1-hadoop1:compile
[INFO] | | +- (com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile - omitted for duplicate)
[INFO] | | - (junit:junit:jar:4.12:test - version managed from 4.11; scope managed from compile; omitted for duplicate)
[INFO] | +- org.apache.hbase:hbase-common:jar:0.98.16.1-hadoop1:compile
[INFO] | | +- (org.apache.hbase:hbase-annotations:jar:0.98.16.1-hadoop1:compile - omitted for duplicate)
......
[INFO] | +- commons-codec:commons-codec:jar:1.7:compile
[INFO] | +- (commons-io:commons-io:jar:2.4:compile - version managed from 2.1; omitted for duplicate)
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- (com.google.guava:guava:jar:18.0:compile - version managed from 12.0.1; omitted for duplicate)
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | +- io.netty:netty:jar:3.10.6.Final:compile
......

Suggested solutions:

Solution: Declare version com.google.protobuf:protobuf-java:jar:2.6.0 as a direct dependency, to override the version 2.5.0 (based on Maven's nearest wins loading strategy).

Thanks.
Best regards,
Coco

关于sqlserver-reader的计划

想问一下关于sqlserver-reader的计划，大概的时间节点

Jetty9.4.15 manager-core模块 branch master ,startup failure

模块：manager-core
jetty:9.4.15
配置好jetty容器，发布manager-core
出现以下异常，涉及类加载流程中，tomcat-util包的冲突问题。最终页面503，不可访问
Caused by: java.util.ServiceConfigurationError: org.apache.juli.logging.Log: Provider org.eclipse.jetty.apache.jsp.JuliLog not a subtype at java.util.ServiceLoader.fail(ServiceLoader.java:239) at java.util.ServiceLoader.access$300(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.juli.logging.LogFactory.<init>(LogFactory.java:78)
请问是jetty版本使用过高吗？

worker启动异常

2019-12-11 17:56:14.777 [main] ERROR com.ucar.datalink.worker.core.boot.WorkerBootStrap - ## Something goes wrong when starting up the datalink worker:
com.ucar.datalink.common.errors.DatalinkException: Worker is not found for client id [null] or ip [10.11.0.176]
at com.ucar.datalink.worker.core.boot.WorkerBootStrap.buildWorkerProps(WorkerBootStrap.java:151) [dl-worker-core-0.0.2.jar:na]
at com.ucar.datalink.worker.core.boot.WorkerBootStrap.boot(WorkerBootStrap.java:52) [dl-worker-core-0.0.2.jar:na]
at com.ucar.datalink.worker.core.boot.WorkerBootStrap.main(WorkerBootStrap.java:198) [dl-worker-core-0.0.2.jar:na]

有哪些公司在使用，有做 统计么？

请问支持mysql5.7.22版本？我想试用一下

请问支持mysql5.7.22版本？我想试用一下

打包正确，启动报错/com/ucar/datalink/manager/core/handler/MediaMappingCha ngeEventHandler.class]: Instantiation of bean failed

能通过idea启动main方法，打开界面

但是打包之后，启动报错
2020-02-26 14:25:52.726 [main] WARN o.s.web.context.support.XmlWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mediaMappingChangeEventHandler' defined in URL [jar:file:/home/zs/datalink/dl-manager/lib/dl-manager-core-1.0.0-beta.jar!/com/ucar/datalink/manager/core/handler/MediaMappingChangeEventHandler.class]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1101) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:663) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:629) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:677) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:548) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:489) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:136) [spring-webmvc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
at javax.servlet.GenericServlet.init(GenericServlet.java:244) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:616) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:396) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:871) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298) [jetty-servlet-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349) [jetty-webapp-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342) [jetty-webapp-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505) [jetty-webapp-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.Server.start(Server.java:387) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.server.Server.doStart(Server.java:354) [jetty-server-9.2.15.v20160210.jar:9.2.15.v20160210]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.2.15.v20160210.jar:9.2.15.v20160210]
at com.ucar.datalink.manager.core.server.JettyServer.startup(JettyServer.java:57) [dl-manager-core-1.0.0-beta.jar:na]
at com.ucar.datalink.manager.core.server.ServerContainer.startup(ServerContainer.java:65) [dl-manager-core-1.0.0-beta.jar:na]
at com.ucar.datalink.manager.core.boot.ManagerBootStrap.boot(ManagerBootStrap.java:38) [dl-manager-core-1.0.0-beta.jar:na]
at com.ucar.datalink.manager.core.boot.ManagerBootStrap.main(ManagerBootStrap.java:28) [dl-manager-core-1.0.0-beta.jar:na]
另外对于数据库密码明文，在idea里面直接使用明文是可行的，打包之后会报密码长度问题

web用户体系能否支持knox或者ldap用户登录？

大数据组件web管理系统太多了，自建用户成本比较高，希望能支持一些标准用户登录体系？

Writer-Hbase 插件抛抛指针

如上图所示，往HBase 写数据时， 抛空指针异常， 可能有并发问题。

t_dl_flinker_job_run_queue表的修改时间(modify_time)默认值不同统一

t_dl_flinker_job_run_queue表的修改时间(modify_time)默认值不同统一,应为
modify_time ： CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT
create_time ： CURRENT_TIMESTAMP COMMENT
初始化DB时使用HS导入，报错。

数据一致性的问题

您好，请问一下：在1.8_深入数据一致性中有提到“数据最终会恢复到最后一个事件对应的状态”，那么是否意味着数据可能会存在短暂的回退现象？比如原本 x=1, x=2, x=3 的操作，最终反映到目的端的时候会出现 x=3, x=1, x=2, x=3 这样的变化现象？Thanks

Writer-Hbase 插件数据为空字符串时，写HBase 失败并抛空指针

writer-hbase -> RdbEventRecordHandler.java 如上图所示，当数据是空字符串时，由于此时不是null, 进行设置后， 在往HBase 写时将失败， 抛空指针异常。

配置RDBMS出错，连接信息都填写正确确提示连不上数据库

网卡流量抛空指针异常

1. 当搜索不到网卡时， 代码中写死“eth0”, 导致空指针异常。

DataLink web admin 介质管理下面的RMDBS模块的建议

端口号应该新增 应该分开读库和写库的让用户自定义端口 要么直接和host写在一起比如:localhost:3306
或者localhost:4406

把maper中的表名称都改为小写，以支持mysql大小写敏感的问题

如标题所述

work 异常

com.ucar.datalink.common.errors.DataLinkException:Worker is not found for client id [null] or ip [172.21.0.1]
环境:
centos7

java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

目前的解决办法是曲线救国：
在work conf设置client.id=1 （此1为Manager的web管理界面的菜单:集群管理->机器管理->机器ID）

执行job的时候提示com.mysql.jdbc.Driver'. This is deprecated.

执行全量任务，postgresql to mysql时候出现如下报错，请问下应该怎样解决？
dl-flinker组件下的admin.log日志里出现

DataxAdminLauncher start job processor failure,please check job config file or start script file !
process failed for job ad_day_data_N6jpZhNMTt.Error Details:
Loading class com.mysql.jdbc.Driver'. This is deprecated. The new driver class is com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.

DataLink能在windows上运行吗？

我在windows 10上运行，进入power shell，执行.\startup.sh，出现CMD界面一瞬即逝。看日志manager.log中只有如下描述：
bash: /c/Program: No such file or directory
bash: /c/Program: No such file or directory

RDBMS MYSQL-> MYSQL 目标数据库没有同步数据

#版本 V1.0.1-beta

1. RDBMS MYSQL -> MYSQL

介质管理-> 数据源建立完毕

增量任务 ->Task 管理->MysqlTask 建立Task 完毕

映射管理 -> 新增映射 完毕

MySql task 启动完毕

2. 向源数据库表插入数据完毕

3. 查看目标表数据没有同步数据。

Manager Log:

2020-01-16 18:05:24.131 [qtp1792384402-147] INFO c.u.d.m.core.web.controller.shadow.TaskShadowController - 收到的参数为:{search={"regex":false,"value":""}, columns=[{"search":{"regex":false,"value":""},"data":"id","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"taskName","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"state","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"mappingIds","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"resetTime","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"createTime","orderable":true,"name":"","searchable":true},{"search":{"regex":false,"value":""},"data":"modifyTime","orderable":true,"name":"","searchable":true}], start=0, length=10, draw=1, state=, taskId=4, order=[{"column":0,"dir":"asc"}]}

Worker Log:

java.lang.NullPointerException: null
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl.getWorkerSystemState(WorkerSystemStateProbeImpl.java:95) ~[dl-worker-core-1.0.0-beta.jar:na]
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl.access$000(WorkerSystemStateProbeImpl.java:31) ~[dl-worker-core-1.0.0-beta.jar:na]
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl$1.run(WorkerSystemStateProbeImpl.java:65) ~[dl-worker-core-1.0.0-beta.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
2020-01-16 18:13:59.130 [Probe-Type-WorkerSystemStateProbe-pool-1] ERROR c.u.d.worker.core.probe.WorkerSystemStateProbeImpl - Worker system state probe failed.
java.lang.NullPointerException: null
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl.getWorkerSystemState(WorkerSystemStateProbeImpl.java:95) ~[dl-worker-core-1.0.0-beta.jar:na]
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl.access$000(WorkerSystemStateProbeImpl.java:31) ~[dl-worker-core-1.0.0-beta.jar:na]
at com.ucar.datalink.worker.core.probe.WorkerSystemStateProbeImpl$1.run(WorkerSystemStateProbeImpl.java:65) ~[dl-worker-core-1.0.0-beta.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_121]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

Task Log:

2020-01-16 17:58:07.404 [Task-5-Reader-reader-mysql] INFO com.ucar.datalink.worker.core.runtime.WorkerTaskReader - TaskReader-reader-mysql finished initialization and start.
2020-01-16 17:58:07.442 [destination = 5-main , address = /172.16.159.139:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> begin to find start position, it will be long time for reset or first position
2020-01-16 17:58:07.443 [destination = 5-main , address = /172.16.159.139:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position just show master status
2020-01-16 17:58:07.444 [destination = 5-main , address = /172.16.159.139:3306 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000013,position=945,serverId=,gtid=,timestamp=] cost : 1ms , the next step is binlog dump

是否有具体是性能指标可以提供参考

任务监控功能不可用

版本：1.0.1
增加任务监控功能试， 前台界面直接抛异常，如下图：

密码加密的建议

我提个建议：我认为默认就是填写明文，程序启动后读取到内存中，然后再改写配置文件成为密文即可，还可以在这基础上加一个配置 是否加密。

no found zk workers ip

为啥我worker启动成功了，执行任务前端报错no found zk workers ip

配置任务和映射完成后，无任何报错，但是目的地表中无数据

问题描述

遇到了数据无法写入的问题。请求帮忙，万分感谢！

配置条件

1. 按照Idea QuickStart一步步启动了Manager和Worker，Worker按照Distributed模式启动

2. 源数据库和目的地数据库均为mysql，且为同一个实例不同的Database

3. 配置了2个介质（数据源），同一个mysql实例不同的数据库，且读库写库配置的完全一致

4. 机器管理显示机器状态正常，后台输出正常

5. 任务列表如下，状态显示正常

   任务ID	任务名称	详情	目标状态	实际状态	所属分组	所属机器	当前日志时间	最近启动时间	操作
   2	test_base_2_1	Details	STARTED	RUNNING	1	1	2019-09-24 16:03:32 (1569312212000)	2019-09-24 16:03:33

6. 映射配置使用白名单，字段完全对应。

这个是基于otter开发的么

重置mysqltask 传递的id 值不对

1. 期望：
   传递正确的id 值
2. 实际：
   传输错误的值，如截图所示

一些优化建议

1. 介质管理 -> RDBMS -> 新增 Etl-Host 不填写时， 会创建失败， 检查代码， 发现是要获取此配置项的值连接数据库 。 如果此为必选项建议强制用户填写。 若非必选项， 建议代码逻辑去除此配置项的连接校验。

2. 增量任务-> Task 管理 -> MysqlTask -> Taks 名称 为例， 当用户填入任意名称时，提示必须以xxx 为前缀， 建议可以写入默认值前缀引导用户填写或鼠标接近输入框时自动提示规则。

3. 启动worker 时， 提示worker/plugin/reader , worker/plugin/writer 等目录不存在， 手动创建目录后， 又提示对应的插件jar 包不存在。 建议可以考虑优化下。 例如在工程打包时预置一些开箱即用的包。