shyiko / mysql-binlog-connector-java Goto Github PK
View Code? Open in Web Editor NEWMySQL Binary Log connector
MySQL Binary Log connector
(This is a question, but I can't apply the question label for some reason).
How to identify which set of events belong to the same mysql transaction?
In EventType.java, there is a comment about the TABLE_MAP event which said that "Row operation events that belong to the same transaction may be grouped into sequences, in which case each such sequence of events begins with a sequence of TABLE_MAP events: one per table used by events in the sequence.". I understood it as we can use TABLE_MAP events as a marker for the beginning of a transaction. However, there's no marker for the end of a transaction. Certainly when seeing a new TABLE_MAP event, that means that the current transaction has ended. But in case of the last happened transaction, how do I process it correctly without an end marker?
hi shyiko,
Thanks for your response.
Sorry I'm new to github, pasted the question to wrong issue. I deleted that one and open a new one for the problem.
The original post is here:
I encountered an exception thrown here. Wondering what caused authenticationResult[0]==(byte)0xFF. The user name&password is definitely correct, mysql server version is 5.0.95-log Source distribution. Is it mysql version related problem?
(new added comment: the code before and at BinaryLogClient:264)
if (authenticationResult[0] != (byte) 0x00 /* ok /) {
if (authenticationResult[0] == (byte) 0xFF / error */) {
byte[] bytes = Arrays.copyOfRange(authenticationResult, 1, authenticationResult.length);
throw new AuthenticationException(new ErrorPacket(bytes).getErrorMessage());
}
throw new AuthenticationException("Unexpected authentication result (" + authenticationResult[0] + ")");
}
The exception I got is:
com.github.shyiko.mysql.binlog.network.AuthenticationException: Unexpected authentication result (-2)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:264)
at com.alipay.xflush.PrometheusTestMain.main(PrometheusTestMain.java:33)
The mysql-binlog-connector version I used is 0.1.0-SNAPSHOT.
I m getting columns number but i want to get columns names any idea?
Is it possible to get the binlog file name corresponding to an Event object? There's the position in the EventHeader, but no file name.
You say in the README "timestamp is timezone-sensitive".
What do you mean, exactly? The bytes in the binary log represent time since the epoch in UTC, right? When they are loaded into a java.sql.Date() on a machine running in the Pacific timezone (GMT-8)... what happens?
I'm attempting to use the library for a project at my job, and I'm running into a problem because the BinaryLogClient
will continue dispatching events to every EventListener
, even if one (or all) of the listeners throws an exception on a previous event.
I have a proposed refactoring in order to make failure handling more flexible. The core ideas (ignoring backward compatibility for a moment, and imagining an ideal world) are:
BinaryLogClient
should only have one EventListener
and one LifecycleListener
.EventListener
throws an exception, the BinaryLogClient
should abort and disconnect immediately.BroadcastEventListener
(and a BroadcastLifecycleListener
) that handles the registration/unregistration logic, dispatches to the registered listeners, and "swallows" the child listeners' exceptions if desired (the same behavior as the current code).I have started a fork exploring these ideas (sacundim@8edbda1). In order not to break existing code that uses the library, I've modified the ideas described above in this way:
BinaryLogClient
should continue to behave exactly the same way as it does now.AbstractBinaryLogClient
and moved most of the original logic there.I still have three issues I'd like to work through, though:
BinaryLogClient.LifecycleListener
has methods that take a BinaryLogClient
argument. In my modified code as of now, this needs to be changed to AbstractBinaryLogClient
, which means that existing LifecycleListener
implementations will no longer compile.I'd appreciate to have your input into this matter.
I had a problem in my replication. the connector said "onCommunicationFailure" with an EventDataDeserializationException caused by a EOFException.
I viewd the source, Maybe there is a bug in AbstractRowsEventDataDeserializer.deserializeRow.
protected Serializable[] deserializeRow(long tableId, BitSet includedColumns, ByteArrayInputStream inputStream)
throws IOException {
TableMapEventData tableMapEvent = tableMapEventByTableId.get(tableId);
byte[] types = tableMapEvent.getColumnTypes();
int[] metadata = tableMapEvent.getColumnMetadata();
BitSet nullColumns = inputStream.readBitSet(types.length, true); <==== THIS LINE
Serializable[] result = new Serializable[numberOfBitsSet(includedColumns)];
}
this is the sql i dumped use mysqlbinlog
# at 397578054
#141204 16:56:25 server id 88 end_log_pos 397578178 CRC32 0xba421d48 Write_rows: table id 51337798 flags: STMT_END_F
BINLOG '
uSGAVBNYAAAAswAAAEaPshcAAEZaDwMAAAEADWNpdHlyZV9iYW90b3UAFllpZWxkX1llYXJNb250
aF9DaXR5XzMANQMDDwUFAwMEBAQEEgUFBQ8FBQQEBAQFBQQEBAQDAw8SAw8SAwMDDwMPDw8DBQQE
AwUEBBIDLzwACAgEBAQEAAgICAkACAgEBAQECAgEBAQElgAAlgAALAEsAR4AHgAIBAQIBAQA/H//
7///D5OtCLU=
uSGAVB5YAAAAfAAAAMKPshcAAEZaDwMAAAEAAgA1f4jDEAAAEACA3gcAAAsAAAACYnQ79Ju92Vq2
QOvVqzkViTBAYQIAALYBAACZlIkOGQBeIJOylwhYQDhw4CigVVVArMdF6gnpS0BxjcUFmpaVQAEA
AAACAAAASB1Cug==
'/*!*/;
### INSERT INTO `cityre_baotou`.`Yield_YearMonth_City_3`
### SET
### @1=2014
### @2=11
### @3='bt'
### @4=5722.8505494567170899
### @5=16.535480122043662021
### @6=609
### @7=438
### @12='2014-12-04 16:56:25'
### @16=''
### @17=96.134258884122999689
### @18=85.337900370767670211
### @23=55.820615085670709732
### @24=1381.6504126422480567
### @29=1
### @53=2
this is the data i dumped from the bin log.
17b28f46 b9 21 80 54 1e 58 00 00 00 7c 00 00 00 c2 8f b2 |.!.T.X...|......|
17b28f56 17 00 00 46 5a 0f 03 00 00 01 00 02 00 35 7f 88 |...FZ........5..|
17b28f66 c3 10 00 00 10 00 80 de 07 00 00 0b 00 00 00 02 |................|
17b28f76 62 74 3b f4 9b bd d9 5a b6 40 eb d5 ab 39 15 89 |bt;[email protected]..|
17b28f86 30 40 61 02 00 00 b6 01 00 00 99 94 89 0e 19 00 |0@a.............|
17b28f96 5e 20 93 b2 97 08 58 40 38 70 e0 28 a0 55 55 40 |^ ....X@8p.(.UU@|
17b28fa6 ac c7 45 ea 09 e9 4b 40 71 8d c5 05 9a 96 95 40 |..E...K@q......@|
17b28fb6 01 00 00 00 02 00 00 00 48 1d 42 ba b9 21 80 54 |........H.B..!.T|
17b28fc6 10 58 00 00 00 1f 00 00 00 e1 8f b2 17 00 00 8c |.X..............|
17b28fd6 e2 63 b3 00 00 00 00 23 70 d7 7f b9 21 80 54 02 |.c.....#p...!.T.|
17b28fe6 58 00 00 00 59 00 00 00 3a 90 b2 17 08 00 e5 43 |X...Y...:......C|
17b28ff6 fd 05 00 00 00 00 0d 00 00 22 00 00 00 00 00 00 |........."......|
17b29006 01 00 00 20 40 00 00 00 00 06 03 73 74 64 04 21 |... @......std.!|
17b29016 00 21 00 21 00 05 06 53 59 53 54 45 4d 63 69 74 |.!.!...SYSTEMcit|
there is 53 fields in my table.the connector should read de 07 00 00 0b 00 00 00 as the data for Field 1 and Field 2,'2014' and '11'.But it read 00 80 de 07 00 00 0b as nullColumns BitSet. So there is not enough bytes for Field Data.It raise EOFException.
Is there somthing wrong with the log event format?
I use this connector in my production environment, Please Help.
Currently, mysql-binlog-connector-java requires that the password is nonempty. If MySQL server account password is empty, mysql-binlog-connector-java won't connect successfully.
I think it is possible to lose binlog events when using the keepalive thread that auto-reconnects upon a disconnect. I remember running into this a while back using v0.1.0. I don't have the time to try to reproduce it right now, though, so I thought I would post it and see if you think it is a problem.
BinaryLogClient.java keeps track of the binlogFilename and binlogPosition of the current event, and updates it after each event is processed:
if (isConnected()) {
notifyEventListeners(event);
updateClientBinlogFilenameAndPosition(event);
updateGtidSet(event);
}
A InnoDB transaction in row-based binlogs has many events: a QUERY event of "BEGIN" to start it off, a TABLE_MAP, WRITE/DELETE/UPDATE events, etc.
If you get disconnected on one of the events in the middle, the keep alive thread will reconnect you at those binlog coordinates. However, in my testing, I noticed that MariaDB 5.5 would not allow you to connect in the middle of a transaction. It would let you connect but would start at the NEXT transaction that appears after your requested coordinates.
Thus, you would skip over events upon reconnecting. You would lose that entire transaction.
Does this sound correct? Again, sorry that I don't have time to attempt a repro right now.
This may be related to #15.
I think readme.md should mention how EventListener#onEvent
is executed.
Is it executed by the network IO thread of mysql-binlog-connector-java? If I have a blocking logic inside onEvent
, will it cause mysql-binlog-connector-java to stop reading new binlog events? Should I run my logic in separate thread that I manage on my own?
I want to write a high performance data replicator, so I don't want to block mysql-binlog-connector-java from reading new binlog events while my replicator logic runs.
Hi,
if any event like WRITE_ROWS, UPDATE_ROWS, DELETE_ROWS event occurs, how can i get database name and table name for which the event occurred?
Please let me know if any way to get those details..
Thanks
I am getting below exception when trying to read the binlog at position given binlog at 52194840
Communication failure,Exception e: java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:182)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:46)
at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:37)
at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:27)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:143)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:595)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:376)
at com.github.shyiko.mysql.binlog.BinaryLogClient$4.run(BinaryLogClient.java:516)
at java.lang.Thread.run(Thread.java:744)
#150728 20:15:14 server id 15836018 end_log_pos 52194840 Xid = 34592233
COMMIT/!/;
#150728 20:15:20 server id 15836018 end_log_pos 52194909 Query thread_id=2267740 exec_time=0 error_code=0
SET TIMESTAMP=1438094720/!/;
BEGIN
/!/;
#150728 20:15:20 server id 15836018 end_log_pos 52195009 Query thread_id=2267740 exec_time=0 error_code=0
SET TIMESTAMP=1438094720/!/;
update TimeTable set Time=1438094720
/!/;
#150728 20:15:20 server id 15836018 end_log_pos 52195036 Xid = 34592241
COMMIT/!/;
QueryEventData qEvent = (QueryEventData) event.getData();
String dbName = qEvent.getDatabase();
if (query.startsWith("create table")|| query.startsWith("CREATE TABLE"))
{
// doing some operations
}
during that time LifecycleListener gets disconnected and again connects from the same position at which create table query occured, again same cycle repeats, it gets struck at this point.
private class BinLogLifeCycleListener implements BinaryLogClient.LifecycleListener
{
public void onConnect(BinaryLogClient client) {
LOG.info("Listener: Binary Log Client connect");//No I18N
}
public void onDisconnect(BinaryLogClient client) {
LOG.info("Listener: Binary Log Client disconnect" );//No I18N
}
}
This happens only for create table query, Here the thing is, the table is not present in master mysql, as it is deleted,
Thanks
Like this
BinaryLogClient client = new BinaryLogClient("hostname", 3306, "username", "password");
client.registerEventListener(new EventListener() {
@Override
public void onEvent(Event event,Acker ack) {
...
ack.ack();// Ack current event
}
});
client.connect();
is this possible ?
Hi,
mysql-binlig-connector-java is really a great tool. By using your tool I am able to get all queries that are executed on master. That really helpful for me, I am also trying to get inserted, updated and deleted rows but i am unable to get that.
Event thought i am have done some INSERT, DELETE and UPDATE i am getting event type as QUERY. Please let me know if I need follow any steps to get rows.
Hello,
I am trying to use this package with scala and I am running into some problems.
Reading from a local binlog file works.
I am using the following code to read from a remote server:
val client = new BinaryLogClient("127.0.0.1", 9797, "user", "pass")
client.registerEventListener(new EventListener() {
def onEvent(event: Event) {
println("New event")
}
})
client.connect
But nothing happens when I run this code.
I have a ssh tunnel on port 9797 forwarding to mysql server. Connecting with the mysqlbinlog
command on that server works.
Any idea on how to debug this? Probably not even a problem of mysql-binlog-connector-java
but I am not sure how to debug.
Thanks.
In the MySQL to MongoDB replication tool Mydit, I continuously save binlog position to MongoDB.
When the replication tool starts, I use BinaryLogClient#setBinlogFilename
and BinaryLogClient#setBinlogPosition` to load the last binlog position (saved in MongoDB from the last run).
If binlog file and position info is too old (the user stops the replication tool, then start it again much later), it seems that BinaryLogClient#connect
doesn't successfully connect to MySQL. It just reconnects periodically, helplessly.
In this case, the replication tool user must manually fix the problem (updating data and updating/reseting binlog info manually). The problem is that BinaryLogClient
doesn't log anything so that the user knows that the binlog file and position info is too old, so he doesn't know that there's problem that he should fix manually.
Is there a method, callback, result code, log etc. to tell if the binlog file and position info is too old?
I can see there is a GtidSet, the format is defined by mysql, but for MariaDB there is an different format described here https://mariadb.com/kb/en/mariadb/global-transaction-id/
The column type&length deserialization code is not the same as MySQL source code.
In the code block of AbstractRowsDeserializer:
the length
variable is set to 0 on the case of typeCode == ColumnType.STRING && meta <= 256
while it is set to meta
in MySQL source.
if (!nullColumns.get(index)) {
int typeCode = types[i] & 0xFF, meta = metadata[i], length = 0;
if (typeCode == ColumnType.STRING.getCode() && meta > 256) {
int meta0 = meta >> 8, meta1 = meta & 0xFF;
if ((meta0 & 0x30) != 0x30) { // long CHAR field
typeCode = meta0 | 0x30;
length = meta1 | (((meta0 & 0x30) ^ 0x30) << 4);
} else {
if (meta0 == ColumnType.SET.getCode() || meta0 == ColumnType.ENUM.getCode() ||
meta0 == ColumnType.STRING.getCode()) {
typeCode = meta0;
length = meta1;
} else {
throw new IOException("Unexpected meta " + meta + " for column of type " + typeCode);
}
}
}
static size_t
log_event_print_value(IO_CACHE *file, const uchar *ptr,
uint type, uint meta,
char *typestr, size_t typestr_length)
{
uint32 length= 0;
if (type == MYSQL_TYPE_STRING)
{
if (meta >= 256)
{
uint byte0= meta >> 8;
uint byte1= meta & 0xFF;
if ((byte0 & 0x30) != 0x30)
{
/* a long CHAR() field: see #37426 */
length= byte1 | (((byte0 & 0x30) ^ 0x30) << 4);
type= byte0 | 0x30;
}
else
length = meta & 0xFF;
}
else
length= meta;
}
//...
I think length = bytes.length >> 2
should be length = bytes.length >> 1
private byte[] reverse(byte[] bytes) {
for (int i = 0, length = bytes.length >> 2; i <= length; i++) {
int j = bytes.length - 1 - i;
byte t = bytes[i];
bytes[i] = bytes[j];
bytes[j] = t;
}
return bytes;
}
If I enable the GTID, can I continue replication by last GTID instead of the binlog file and log position ?
hi again,
I want to report a new issue. During my using of mysql-binlog-connector-java 0.1.0-snapshot, sometimes I will find this exception. I found you thrown the EOF in the code. Why do that? If connection is set to keep alive, isn't it WAD when reaching EOF of binlog?
java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:180)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:46)
at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:35)
at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:27)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:98)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:465)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:298)
Here is where the exception was thrown:
public int read() throws IOException {
int result;
if (peek == null) {
result = readWithinBlockBoundaries();
} else {
result = peek;
peek = null;
}
if (result == -1) {
throw new EOFException();
}
return result;
}
BTW:it happened when replicating with several mysql servers, and for each of the servers, the exception might not happen sometimes even no configuration was changed for those machine as for as I remember. Those servers include 5.6.x, on which happened once, and version 5.1.28, on which happened very frequently. However, when I caught the exception and reconnected again, sometimes connecting was success. From what I saw, the exception happened when just connecting to server as well as connecting to server and waiting a long time without no binlog events. I also observed the exception came after immediately FORMAT_DESCRIPTION event was parsed. However, if reaching EOF of binlog is the condition of this exception, why it did not happen all the time?
Related to #28 since the library depends on those being deserialized to a specific type.
Stack Trace
Exception:
com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1433181420000, eventType=UPDATE_ROWS, serverId=478847, headerLength=19, dataLength=183, nextPosition=38149950, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:140)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:116)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:501)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:328)
at com.github.shyiko.mysql.binlog.BinaryLogClient$4.run(BinaryLogClient.java:425)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at com.github.shyiko.mysql.binlog.io.BufferedSocketInputStream.read(BufferedSocketInputStream.java:51)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readWithinBlockBoundaries(ByteArrayInputStream.java:192)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:174)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readLong(ByteArrayInputStream.java:57)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeCell(AbstractRowsEventDataDeserializer.java:121)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:96)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserializeRows(UpdateRowsEventDataDeserializer.java:72)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:58)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:33)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:134)
... 5 more
Using 0.1.1 version of mysql-binlog-connector-java
public void onCommunicationFailure(BinaryLogClient client, Exception ex) {
System.out.println("Communication failure ,Exception: "+ex.printStackTrace());
}
I'm getting above exception when any update/write/delete event occurs, saying not able to deserialize the event data.
As you suggested, I tried setting BinaryLogClient.setServerId(uniqueid) and also keepAliveInterval to 60 sec, but still exception persists
EventDeserializer::deserializeEventData
Hi
Locally Iam storing binlog details(name, position, serverid etc.,), to read binlogs from db cluster(i.e., master db).
If suppose any failover happens, and client connect to db cluster (but now to slave i.e., current master now), but the bin log details doesn't match with what we have locally. And when we try to connect with this, it get blocked in connect() method saying (connected to clusterip:port wit bin log name and position is bin.0000XX and YYYYYYY respectively). Please help me to handle the client connect when failover happens.
Thanks
Akram Syed
Every time I try to connect to any master (mysql 5.5.6) I get the following error:
java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:180)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readZeroTerminatedString(ByteArrayInputStream.java:81)
at com.github.shyiko.mysql.binlog.network.protocol.GreetingPacket.(GreetingPacket.java:48)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:297)
at mem_search.Go.main(Go.java:23)
I've tried multiple database server and I tried remote connect as well as running it on the local host. Can't seem to get it to work. Any tips?
I'm in the America/Los_Angeles timezone. I ran into these two problems while running the integration test. First: the test setup fails:
Tests run: 38, Failures: 1, Errors: 0, Skipped: 33, Time elapsed: 1.235 sec <<< FAILURE! - in TestSuite
setUp(com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest) Time elapsed: 0.593 sec <<< FAILURE!
java.sql.SQLException: Unknown or incorrect time zone: '--7:00'
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2713)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2663)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:888)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:730)
at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection$1.execute(BinaryLogClientIntegrationTest.java:565)
at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection$1.execute(BinaryLogClientIntegrationTest.java:556)
at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection.execute(BinaryLogClientIntegrationTest.java:574)
at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection.<init>(BinaryLogClientIntegrationTest.java:556)
at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection.<init>(BinaryLogClientIntegrationTest.java:539)
at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest.setUp(BinaryLogClientIntegrationTest.java:87)
The code to generate the timezone string for the SET time_zone
command is wrong, and incorrectly generates two minus signs for those of us who live in the past (SET time_zone = '--7:00'
). Commit 6eaa9a3 in my fork gets past this failure, but I'm not sure yet that this is the right fix (see below).
After applying that change, however, I do run into a different failure, seemingly related to timezones and daylight saving time:
Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.325 sec <<< FAILURE! - in TestSuite
testDeserializationOfDifferentColumnTypes(com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest) Time elapsed: 0.777 sec <<< FAILURE!
java.lang.AssertionError: Lists differ at element [0]: 1989-03-18 01:02:03.0 != 1989-03-18 00:02:03.0 expected [1989-03-18 01:02:03.0] but found [1989-03-18 00:02:03.0]
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.failNotEquals(Assert.java:494)
at org.testng.Assert.assertEquals(Assert.java:123)
at org.testng.Assert.assertEquals(Assert.java:549)
at org.testng.Assert.assertEquals(Assert.java:668)
at org.testng.Assert.assertEquals(Assert.java:725)
at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest.testDeserializationOfDifferentColumnTypes(BinaryLogClientIntegrationTest.java:206)
If I modify the test case to '1989-05-18 01:02:03.0'
, then it passes. This likely has some connection to the SET time_zone = '-7:00'
, because on 1989-03-18 01:02:03.0, California was at a -8:00 offset, whereas today (2014-04-18) we're -7:00. So there's the chance as well that this test could variably succeed or fail depending on the season.
This may solve a lot of problem, like #15 ,for currently Listener, I have to use a BlockingQueue to implement some pull style code.
I would like to publish the raw event data, and then parse the event data on the subscribers to let them handle the events themselves.
Hello,
I am playing around with mysql-binlog-connector-java and I am failing to understand how exactly do I loop and get the values from a WriteRowsEventData (or UpdatedRowsEventData).
Let's say we got the Event inside onEvent(Event event)...what's next?
In open-replicator you were doing row.getColumns() and you got the column values.
It's not very clear what to do after you got the rows and how to deserialize (or does this happen automatically?)
Cheers
How to get notification for tables having fk realtionship like on delete cascade etc..
Hi.
This is not a problem about 'mysql-binlog-connector-java'.
I just find out that there is no field name in TableMapEventData. how can i regenerate the SQL from WRITE_ROWS,UPDATE_ROWS,DELETE_ROWS.or there is other method to update slave mysql database?
Thanks for help.
distributed processing mysql binlog byte data but not deserializer it.
not deserializer bytes. only distributed .
sorrry for my bad english.
thank you.
In TableMapEventData
I can get DB name, table name, column types etc. But there's no info about column names.
How to get column names of a table? Is there a feature in mysql-binlog-connector-java to do that? Or must I use other tools like JDBC? It's very helpful if there's instruction about this in the README.
(Sorry this is a question. I don't know a better place to post it so I post it here.)
Any chance I can get some instructions about obtaining the row values before and after a modification in case of RBR ?
I couldn't find any documentation about this.
Hi,
First of all, mysql-binlog-connector-java is a really great tool. We use this to create a test environment by replicating the production database in a special way. The connector works great but we experienced an increasing lag behind the master server even without any event listeners. The connection between the servers are good and we do not have lag with mysql's master-slave replication.
Do you have any idea how to improve the processing time somehow? What do you think what causes the lag?
Hi,
mysql-binlog-connector-java is a great tool. I have tried some other mysql replication tools, like Databus(LinkedIn), canal(alibaba).They all need to set binlog_format=ROW.
While using mysql-binlog-connector-java, I tried 'MIXED' and 'ROW'. Both can work.
I got 'WRITE_ROWS' event while using 'ROW' format. 'QUERY' event while using 'MIXED' format.It seems both format are supported.
Is it OK to use binlog_format='MIXED' and mysql-binlog-connector-java, in my production environment?
thanks.
Hello,
I am looking at the BinaryLogClient and I realize that you can not set a server id but it fetches automatically the id of the master which I believe it's not the right behavior.
A unique server id must be set for master and all replication servers as you can also read here:
http://dev.mysql.com/doc/refman/5.0/en/replication-options-slave.html
Server ID. On the master and each slave, you must use the server-id option to establish
a unique replication ID in the range from 1 to 232 – 1. “Unique” means that each ID must
be different from every other ID in use by any other replication master or slave.
Am I missing something?
Cheers
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.