actiontech/dtle

Distributed Data Transfer Service for MySQL

dtle's Issues

[INTERNAL USE]

[INTERNAL USE]

delete a job, add another job, new job won't work

Description

Steps to reproduce the issue:

1. add job1 (57:3308-> 48:3307)
2. delete job1 (57:3308-> 48:3307)
3. post job2 (56:3308->48:3307)

Describe the results you received:
job2 replication doesn't work

Describe the results you expected:
job2 replication should work

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

pause working job will error

Description

Steps to reproduce the issue:

1. set up a replication called job1
2. make sure the replication working
3. pause job1

Describe the results you received:
dtle exit with error:

[ERR] mysql.extractor: unexpected error on publish, got nats: connection closed.
...
[ERR] mysql.applier: exec set gtid_next err: sql: database is closed
[ERR] mysql.applier: unexpected err: sql: database is closed

Describe the results you expected:
it should work

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

full-data replication with character "`" report error

Description

Steps to reproduce the issue:

1. set up a full-data replication
2. execute sql in master

insert into varchar_columns (id,c_varchar) values (6,'~`!@#$%&*()-=_+{}[]|:;<>,.?/');

Describe the results you received:
dtle report SQL syntax error

Describe the results you expected:

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

[INTERNAL USE]

[INTERNAL USE]

rpm should support relocation

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
rpm should support relocation (http://rpm5.org/docs/api/relocatable.html).

rpm -ivh --prefix /opt/dtle dtle-1.0.0-qa.x86_64.rpm should not failed

Describe alternatives you've considered
No

[INTERNAL USE]

[INTERNAL USE]

network limit

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
Add a parameter called "network_ratelimit" on job, to limit network resource usage.

Describe alternatives you've considered
None

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

connection to MySQL leaks after job deleted

Description

Steps to reproduce the issue:

1. set up a replication job
2. delete the job
3. execute show processlist on source

Describe the results you received:
Found connection to MySQL leaks

Describe the results you expected:

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

`grant all` error

Description

Steps to reproduce the issue:

1. set up a replication
2. exec grant all xxx in source

Describe the results you received:
dtle error:

parse query [GRANT ALL PRIVILEGES xxx] event failed ...

Describe the results you expected:
no error

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

migrate n-to-1, multi master with large amount of data, dtle report error

Description

Steps to reproduce the issue:

1. prepare large amount of data in masters
2. set up a n-to-1 replication

Describe the results you received:
dtle report error:

[ERR] mysql.extractor: Error in Request: nats: timeout

Describe the results you expected:

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

fail to start when source has SQL which update user's password

Description

Steps to reproduce the issue:

1. ALTER USER xxx in source database
2. start dtle with a replication task

Describe the results you received:
dtle failed to start

Describe the results you expected:
replication should work

Output of ./dtle version:

(Not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

create view cause error

Description

Steps to reproduce the issue:

1. set up a replication
2. execute CREATE VIEW v AS SELECT count(*) from sbtest1

Describe the results you received:

dtle report error:

[INFO] mysql.extractor: streamEvents encountered unexpected error: parse query [CREATE ... VIEW v AS SELECT count(*) from sbtest1" (total length 107)

Describe the results you expected:
no error

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

full data replication, db without table is not replicated

Description

Steps to reproduce the issue:

1. prepare an empty database in master
2. set up a full-data replication

Describe the results you received:

1. replication works fine
2. the empty database is not found in slave

Describe the results you expected:
empty database should be replicated

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

full-data replication doesn't work for table with foreign keys

Description

Steps to reproduce the issue:

1. create table on both sides, disable foreign key check
2. set up a replication

Describe the results you received:
dtle reports it cannot replicate table with foreign key

Describe the results you expected:

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

migrate n-to-1, got error when replay `create index`

Description

Steps to reproduce the issue:

1. set up a 2-to-1 replication
2. execute create index k_1 on sbtest1(K) in both two sources

Describe the results you received:
dtle error:

[ERR] mysql.applier: exec sql error: Error 1061: Duplicate key name 'k_1'

Describe the results you expected:
No error

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

migrate n-to-1, got error `No database selected` after a `drop`

Description

Steps to reproduce the issue:

1. execute drop database if exists sbtest in both sources
2. set up a 2-to-1 replication
3. execute

create database if not exists sbtest;
USE sbtest;
CREATE TABLE sbtest1 (
id int(10) unsigned NOT NULL,
k int(10) unsigned NOT NULL DEFAULT '0',
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 MAX_ROWS=1000000

Describe the results you received:
dtle error:

[ERR] mysql.applier: exec sql error: Error 1046: No database selected

Describe the results you expected:
it should work

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

target send 15%+ data to source, which doesn't make sense

Description

Steps to reproduce the issue:

1. set up a replication
2. monitor on network

Describe the results you received:

1. found target send 15%+ of data back to source, which doesn't make sense

Describe the results you expected:

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

auto GTID mode

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like
if set GTID to "AUTO", like MySQL do, target should detect source's GTID and start replication from it

Describe alternatives you've considered
None

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

metric name is not human-friendly

Description

Steps to reproduce the issue:

1. metric name is like dtle_test74_client_allocs_job1500516500_Dest_6f7b1172_9776_610f_b887_e460f05bcecf_Dest_table_dest_queue_size

Describe the results you received:

1. we suggest metric to be udp_dest_queue_size, and attributes should be put into labels

Describe the results you expected:

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

data with escape character report error

Description

Steps to reproduce the issue:

1. set up a replication
2. execute insert into Dtypes_string_blob (sid,num,col1,col2,col3,col4) values ('ERNSH',1,'123','abc','__%%%\','a_bcdefg');

Describe the results you received:
dtle report error:

Describe the results you expected:

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

log refine

Description

Steps to reproduce the issue:

1. add/delete job has no log
2. there is no log key to distinguish the log

Describe the results you received:

1. log should be refined

Describe the results you expected:

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

[INTERNAL USE]

[INTERNAL USE]

replication not work after slave restart

Description

Steps to reproduce the issue:

1. set up a replication
2. kill slave
3. insert data in master
4. restart slave

Describe the results you received:
replication not work

Describe the results you expected:
replication should work
Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

Add support for MySQL GTIDs

Is your feature request related to a problem? Please describe.
No

Describe the solution you'd like

• Describe the feature in general

GTIDs are implemented as a new binlog event type. GTIDs are written to the
binlog prior to the transaction that they describe. They effectivity represent
a new type of commit position. MySQL provides a new sessions variable
NEXT_GTID that can be used to set the GTID when replaying transactions in
slaves.

dtle will parse GTIDs correctly, set them correctly on slaves, and will
take advantage of them to prevent multi-master loops.

• Describe the feature interface

GTIDs have a number of uses in dtle that should be includes in the
implementation.

Parse GTID events and add them to the event log.
Set GTID values on the slave.
Optionally use GTID values to prevent replication loops as a replacement
from the current system using MySQL binlog and offset values to identify the
position of slaves.

• Why is this important

GTIDs if usable provide a means of avoiding replication loops.They also make
it possible switch between different MySQL.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

delete job failed

Description

Steps to reproduce the issue:

1. delete a job via API

Describe the results you received:
No job deleted

Describe the results you expected:
Specific job should be deleted

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

[INTERNAL USE]

migrate n-to-1, got error when execute `ALTER TABLE ... ADD ... FOREIGH KEY ...`

Description

Steps to reproduce the issue:

1. set up n-to-1 replication
2. execute ALTER TABLE sbtest2 ADD CONSTRAINT fk_PerOrders FOREIGN KEY (k) REFERENCES sbtest1(k); in source

Describe the results you received:
dtle error:

[ERR] mysql.applier: unspected error: Error 1022: Can't write; duplicate key in table '#sql-4641_310'

Describe the results you expected:
it should work

Output of ./dtle version:

(not available)

Additional information you deem important (e.g. issue happens only occasionally):

Additional details (log, config, job config etc):