shyiko/mysql-binlog-connector-java

MySQL Binary Log connector

mysql-binlog-connector-java's Issues

How to identify which set of events belong to the same mysql transaction?

(This is a question, but I can't apply the question label for some reason).

How to identify which set of events belong to the same mysql transaction?

In EventType.java, there is a comment about the TABLE_MAP event which said that "Row operation events that belong to the same transaction may be grouped into sequences, in which case each such sequence of events begins with a sequence of TABLE_MAP events: one per table used by events in the sequence.". I understood it as we can use TABLE_MAP events as a marker for the beginning of a transaction. However, there's no marker for the end of a transaction. Certainly when seeing a new TABLE_MAP event, that means that the current transaction has ended. But in case of the last happened transaction, how do I process it correctly without an end marker?

(MySQL 5.0.95) AuthenticationException: Unexpected authentication result (-2)

hi shyiko,

Thanks for your response.

Sorry I'm new to github, pasted the question to wrong issue. I deleted that one and open a new one for the problem.

The original post is here:
I encountered an exception thrown here. Wondering what caused authenticationResult[0]==(byte)0xFF. The user name&password is definitely correct, mysql server version is 5.0.95-log Source distribution. Is it mysql version related problem?

(new added comment: the code before and at BinaryLogClient:264)
if (authenticationResult[0] != (byte) 0x00 /* ok /) {
if (authenticationResult[0] == (byte) 0xFF / error */) {
byte[] bytes = Arrays.copyOfRange(authenticationResult, 1, authenticationResult.length);
throw new AuthenticationException(new ErrorPacket(bytes).getErrorMessage());
}
throw new AuthenticationException("Unexpected authentication result (" + authenticationResult[0] + ")");
}

The exception I got is:

com.github.shyiko.mysql.binlog.network.AuthenticationException: Unexpected authentication result (-2)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:264)
at com.alipay.xflush.PrometheusTestMain.main(PrometheusTestMain.java:33)

The mysql-binlog-connector version I used is 0.1.0-SNAPSHOT.

How to get columns names?

I m getting columns number but i want to get columns names any idea?

How to get the binlog file name corresponding to an Event?

Is it possible to get the binlog file name corresponding to an Event object? There's the position in the EventHeader, but no file name.

"timestamp is timezone-sensitive"?

You say in the README "timestamp is timezone-sensitive".

What do you mean, exactly? The bytes in the binary log represent time since the epoch in UTC, right? When they are loaded into a java.sql.Date() on a machine running in the Pacific timezone (GMT-8)... what happens?

BinaryLogClient will always continue even if one of the EventListeners errors out on an event

I'm attempting to use the library for a project at my job, and I'm running into a problem because the BinaryLogClient will continue dispatching events to every EventListener, even if one (or all) of the listeners throws an exception on a previous event.

I have a proposed refactoring in order to make failure handling more flexible. The core ideas (ignoring backward compatibility for a moment, and imagining an ideal world) are:

1. BinaryLogClient should only have one EventListener and one LifecycleListener.
2. When the one EventListener throws an exception, the BinaryLogClient should abort and disconnect immediately.
3. The multiple listener behavior in the current library should be factored out into a BroadcastEventListener (and a BroadcastLifecycleListener) that handles the registration/unregistration logic, dispatches to the registered listeners, and "swallows" the child listeners' exceptions if desired (the same behavior as the current code).

I have started a fork exploring these ideas (sacundim@8edbda1). In order not to break existing code that uses the library, I've modified the ideas described above in this way:

• BinaryLogClient should continue to behave exactly the same way as it does now.
• I've added a class AbstractBinaryLogClient and moved most of the original logic there.

I still have three issues I'd like to work through, though:

1. My changes so far still break compatibility with existing code, because BinaryLogClient.LifecycleListener has methods that take a BinaryLogClient argument. In my modified code as of now, this needs to be changed to AbstractBinaryLogClient, which means that existing LifecycleListener implementations will no longer compile.
2. I'm still generally just not certain that I have the best factoring.
3. I haven't run any tests yet.

I'd appreciate to have your input into this matter.

It should be possible to specify custom keep alive strategy

EventDataDeserializationException on EXT_WRITE_ROWS Event

I had a problem in my replication. the connector said "onCommunicationFailure" with an EventDataDeserializationException caused by a EOFException.

I viewd the source, Maybe there is a bug in AbstractRowsEventDataDeserializer.deserializeRow.

 protected Serializable[] deserializeRow(long tableId, BitSet includedColumns, ByteArrayInputStream inputStream)
            throws IOException {
        TableMapEventData tableMapEvent = tableMapEventByTableId.get(tableId);
        byte[] types = tableMapEvent.getColumnTypes();
        int[] metadata = tableMapEvent.getColumnMetadata();
        BitSet nullColumns = inputStream.readBitSet(types.length, true);         <====   THIS LINE
        Serializable[] result = new Serializable[numberOfBitsSet(includedColumns)];
}

this is the sql i dumped use mysqlbinlog

# at 397578054
#141204 16:56:25 server id 88  end_log_pos 397578178 CRC32 0xba421d48   Write_rows: table id 51337798 flags: STMT_END_F

BINLOG '
uSGAVBNYAAAAswAAAEaPshcAAEZaDwMAAAEADWNpdHlyZV9iYW90b3UAFllpZWxkX1llYXJNb250
aF9DaXR5XzMANQMDDwUFAwMEBAQEEgUFBQ8FBQQEBAQFBQQEBAQDAw8SAw8SAwMDDwMPDw8DBQQE
AwUEBBIDLzwACAgEBAQEAAgICAkACAgEBAQECAgEBAQElgAAlgAALAEsAR4AHgAIBAQIBAQA/H//
7///D5OtCLU=
uSGAVB5YAAAAfAAAAMKPshcAAEZaDwMAAAEAAgA1f4jDEAAAEACA3gcAAAsAAAACYnQ79Ju92Vq2
QOvVqzkViTBAYQIAALYBAACZlIkOGQBeIJOylwhYQDhw4CigVVVArMdF6gnpS0BxjcUFmpaVQAEA
AAACAAAASB1Cug==
'/*!*/;
### INSERT INTO `cityre_baotou`.`Yield_YearMonth_City_3`
### SET
###   @1=2014
###   @2=11
###   @3='bt'
###   @4=5722.8505494567170899
###   @5=16.535480122043662021
###   @6=609
###   @7=438
###   @12='2014-12-04 16:56:25'
###   @16=''
###   @17=96.134258884122999689
###   @18=85.337900370767670211
###   @23=55.820615085670709732
###   @24=1381.6504126422480567
###   @29=1
###   @53=2

this is the data i dumped from the bin log.

17b28f46 b9 21 80 54 1e 58 00 00 00 7c 00 00 00 c2 8f b2 |.!.T.X...|......|
17b28f56 17 00 00 46 5a 0f 03 00 00 01 00 02 00 35 7f 88 |...FZ........5..|
17b28f66 c3 10 00 00 10 00 80 de 07 00 00 0b 00 00 00 02 |................|
17b28f76 62 74 3b f4 9b bd d9 5a b6 40 eb d5 ab 39 15 89 |bt;[email protected]..|
17b28f86 30 40 61 02 00 00 b6 01 00 00 99 94 89 0e 19 00 |0@a.............|
17b28f96 5e 20 93 b2 97 08 58 40 38 70 e0 28 a0 55 55 40 |^ ....X@8p.(.UU@|
17b28fa6 ac c7 45 ea 09 e9 4b 40 71 8d c5 05 9a 96 95 40 |..E...K@q......@|
17b28fb6 01 00 00 00 02 00 00 00 48 1d 42 ba b9 21 80 54 |........H.B..!.T|
17b28fc6 10 58 00 00 00 1f 00 00 00 e1 8f b2 17 00 00 8c |.X..............|
17b28fd6 e2 63 b3 00 00 00 00 23 70 d7 7f b9 21 80 54 02 |.c.....#p...!.T.|
17b28fe6 58 00 00 00 59 00 00 00 3a 90 b2 17 08 00 e5 43 |X...Y...:......C|
17b28ff6 fd 05 00 00 00 00 0d 00 00 22 00 00 00 00 00 00 |........."......|
17b29006 01 00 00 20 40 00 00 00 00 06 03 73 74 64 04 21 |... @......std.!|
17b29016 00 21 00 21 00 05 06 53 59 53 54 45 4d 63 69 74 |.!.!...SYSTEMcit|

there is 53 fields in my table.the connector should read de 07 00 00 0b 00 00 00 as the data for Field 1 and Field 2,'2014' and '11'.But it read 00 80 de 07 00 00 0b as nullColumns BitSet. So there is not enough bytes for Field Data.It raise EOFException.

Is there somthing wrong with the log event format?

I use this connector in my production environment, Please Help.

Empty password is not supported

Currently, mysql-binlog-connector-java requires that the password is nonempty. If MySQL server account password is empty, mysql-binlog-connector-java won't connect successfully.

Decrease memory consumption by tweaking EventDeserializer.nextEvent

Support mysqlbinlog-style log requesting (to-last-log)

http://dev.mysql.com/doc/refman/5.6/en/mysqlbinlog-server-id.html
http://dev.mysql.com/doc/internals/en/generic-response-packets.html#packet-EOF_Packet

Possible to lose events when using keep alive

I think it is possible to lose binlog events when using the keepalive thread that auto-reconnects upon a disconnect. I remember running into this a while back using v0.1.0. I don't have the time to try to reproduce it right now, though, so I thought I would post it and see if you think it is a problem.

BinaryLogClient.java keeps track of the binlogFilename and binlogPosition of the current event, and updates it after each event is processed:

                if (isConnected()) {
                    notifyEventListeners(event);
                    updateClientBinlogFilenameAndPosition(event);
                    updateGtidSet(event);
                }

A InnoDB transaction in row-based binlogs has many events: a QUERY event of "BEGIN" to start it off, a TABLE_MAP, WRITE/DELETE/UPDATE events, etc.

If you get disconnected on one of the events in the middle, the keep alive thread will reconnect you at those binlog coordinates. However, in my testing, I noticed that MariaDB 5.5 would not allow you to connect in the middle of a transaction. It would let you connect but would start at the NEXT transaction that appears after your requested coordinates.

Thus, you would skip over events upon reconnecting. You would lose that entire transaction.

Does this sound correct? Again, sorry that I don't have time to attempt a repro right now.

How EventListener#onEvent is executed?

This may be related to #15.

I think readme.md should mention how EventListener#onEvent is executed.

Is it executed by the network IO thread of mysql-binlog-connector-java? If I have a blocking logic inside onEvent, will it cause mysql-binlog-connector-java to stop reading new binlog events? Should I run my logic in separate thread that I manage on my own?

I want to write a high performance data replicator, so I don't want to block mysql-binlog-connector-java from reading new binlog events while my replicator logic runs.

Add a change log (use olivierlacan/keep-a-changelog as a guidance)

Publish to Maven central (make sure sources & javadocs included)

How to get DBName and TableName in WriteRowsEvent/UpdateRowsEvent/DeleteRowsEvent

Hi,

if any event like WRITE_ROWS, UPDATE_ROWS, DELETE_ROWS event occurs, how can i get database name and table name for which the event occurred?

Please let me know if any way to get those details..

Thanks

EOFException while reading binlog

I am getting below exception when trying to read the binlog at position given binlog at 52194840
Communication failure,Exception e: java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:182)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:46)
at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:37)
at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:27)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:143)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:595)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:376)
at com.github.shyiko.mysql.binlog.BinaryLogClient$4.run(BinaryLogClient.java:516)
at java.lang.Thread.run(Thread.java:744)

at 52194813

#150728 20:15:14 server id 15836018 end_log_pos 52194840 Xid = 34592233

COMMIT/!/;

at 52194840

#150728 20:15:20 server id 15836018 end_log_pos 52194909 Query thread_id=2267740 exec_time=0 error_code=0

SET TIMESTAMP=1438094720/!/;
BEGIN
/!/;

at 52194909

#150728 20:15:20 server id 15836018 end_log_pos 52195009 Query thread_id=2267740 exec_time=0 error_code=0

SET TIMESTAMP=1438094720/!/;
update TimeTable set Time=1438094720
/!/;

at 52195009

#150728 20:15:20 server id 15836018 end_log_pos 52195036 Xid = 34592241

COMMIT/!/;

BinaryLogClient Disconnects and connects again when create table query comes for the table which is not available(deleted in master mysql)

QueryEventData qEvent = (QueryEventData) event.getData();
String dbName = qEvent.getDatabase();
if (query.startsWith("create table")|| query.startsWith("CREATE TABLE"))
{
// doing some operations
}

during that time LifecycleListener gets disconnected and again connects from the same position at which create table query occured, again same cycle repeats, it gets struck at this point.

private class BinLogLifeCycleListener implements BinaryLogClient.LifecycleListener
{
public void onConnect(BinaryLogClient client) {
LOG.info("Listener: Binary Log Client connect");//No I18N
}

           public void onDisconnect(BinaryLogClient client) {
        LOG.info("Listener: Binary Log Client disconnect" );//No I18N
    }

}

This happens only for create table query, Here the thing is, the table is not present in master mysql, as it is deleted,

Thanks

Any semi-sync replication support?

Like this

BinaryLogClient client = new BinaryLogClient("hostname", 3306, "username", "password");
client.registerEventListener(new EventListener() {

    @Override
    public void onEvent(Event event,Acker ack) {
        ...
        ack.ack();// Ack current event
    }
});
client.connect();

is this possible ?

Do not try (on push) to deploy SNAPSHOT build to "Sonatype Nexus Snapshots" unless it's upstream\master branch

Trying to get inserted, updated and deleted rows, but i am unable to get

Hi,

mysql-binlig-connector-java is really a great tool. By using your tool I am able to get all queries that are executed on master. That really helpful for me, I am also trying to get inserted, updated and deleted rows but i am unable to get that.

Event thought i am have done some INSERT, DELETE and UPDATE i am getting event type as QUERY. Please let me know if I need follow any steps to get rows.

No events received when using `mysql-binlog-connector-java` with scala

Hello,

I am trying to use this package with scala and I am running into some problems.

Reading from a local binlog file works.

I am using the following code to read from a remote server:

val client = new BinaryLogClient("127.0.0.1", 9797, "user", "pass")

client.registerEventListener(new EventListener() {
  def onEvent(event: Event) {
    println("New event")
  }
})

client.connect

But nothing happens when I run this code.

I have a ssh tunnel on port 9797 forwarding to mysql server. Connecting with the mysqlbinlog command on that server works.

Any idea on how to debug this? Probably not even a problem of mysql-binlog-connector-java but I am not sure how to debug.

Thanks.

How to know if binlog file and position info is too old?

In the MySQL to MongoDB replication tool Mydit, I continuously save binlog position to MongoDB.

When the replication tool starts, I use BinaryLogClient#setBinlogFilename and BinaryLogClient#setBinlogPosition` to load the last binlog position (saved in MongoDB from the last run).

If binlog file and position info is too old (the user stops the replication tool, then start it again much later), it seems that BinaryLogClient#connect doesn't successfully connect to MySQL. It just reconnects periodically, helplessly.

In this case, the replication tool user must manually fix the problem (updating data and updating/reseting binlog info manually). The problem is that BinaryLogClient doesn't log anything so that the user knows that the binlog file and position info is too old, so he doesn't know that there's problem that he should fix manually.

Is there a method, callback, result code, log etc. to tell if the binlog file and position info is too old?

Support MariaDB flavor GTID

I can see there is a GtidSet, the format is defined by mysql, but for MariaDB there is an different format described here https://mariadb.com/kb/en/mariadb/global-transaction-id/

AbstractRowsDeserializer issue

The column type&length deserialization code is not the same as MySQL source code.

In the code block of AbstractRowsDeserializer:
the length variable is set to 0 on the case of typeCode == ColumnType.STRING && meta <= 256
while it is set to meta in MySQL source.

            if (!nullColumns.get(index)) {
                int typeCode = types[i] & 0xFF, meta = metadata[i], length = 0;
                if (typeCode == ColumnType.STRING.getCode() && meta > 256) {
                    int meta0 = meta >> 8, meta1 = meta & 0xFF;
                    if ((meta0 & 0x30) != 0x30) { // long CHAR field
                        typeCode = meta0 | 0x30;
                        length = meta1 | (((meta0 & 0x30) ^ 0x30) << 4);
                    } else {
                        if (meta0 == ColumnType.SET.getCode() || meta0 == ColumnType.ENUM.getCode() ||
                                meta0 == ColumnType.STRING.getCode()) {
                            typeCode = meta0;
                            length = meta1;
                        } else {
                            throw new IOException("Unexpected meta " + meta + " for column of type " + typeCode);
                        }
                    }
                }

static size_t
log_event_print_value(IO_CACHE *file, const uchar *ptr,
                      uint type, uint meta,
                      char *typestr, size_t typestr_length)
{
  uint32 length= 0;

  if (type == MYSQL_TYPE_STRING)
  {
    if (meta >= 256)
    {
      uint byte0= meta >> 8;
      uint byte1= meta & 0xFF;

      if ((byte0 & 0x30) != 0x30)
      {
        /* a long CHAR() field: see #37426 */
        length= byte1 | (((byte0 & 0x30) ^ 0x30) << 4);
        type= byte0 | 0x30;
      }
      else
        length = meta & 0xFF;
    }
    else
      length= meta;
  }
//...

"binary" data deserialization produces incorrect value unless default charset is ISO-8859-1

A bug in ByteArrayInputStream

I think length = bytes.length >> 2 should be length = bytes.length >> 1

    private byte[] reverse(byte[] bytes) {
        for (int i = 0, length = bytes.length >> 2; i <= length; i++) {
            int j = bytes.length - 1 - i;
            byte t = bytes[i];
            bytes[i] = bytes[j];
            bytes[j] = t;
        }
        return bytes;
    }

How can I restore binlog position with a GTID ?

If I enable the GTID, can I continue replication by last GTID instead of the binlog file and log position ?

java.io.EOFException at ByteArrayInputStream.java:180

hi again,

I want to report a new issue. During my using of mysql-binlog-connector-java 0.1.0-snapshot, sometimes I will find this exception. I found you thrown the EOF in the code. Why do that? If connection is set to keep alive, isn't it WAD when reaching EOF of binlog?

java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:180)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readInteger(ByteArrayInputStream.java:46)
at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:35)
at com.github.shyiko.mysql.binlog.event.deserialization.EventHeaderV4Deserializer.deserialize(EventHeaderV4Deserializer.java:27)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:98)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:465)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:298)

Here is where the exception was thrown:

public int read() throws IOException {
    int result;
    if (peek == null) {
        result = readWithinBlockBoundaries();
    } else {
        result = peek;
        peek = null;
    }
    if (result == -1) {
        throw new EOFException();
    }
    return result;
}

BTW:it happened when replicating with several mysql servers, and for each of the servers, the exception might not happen sometimes even no configuration was changed for those machine as for as I remember. Those servers include 5.6.x, on which happened once, and version 5.1.28, on which happened very frequently. However, when I caught the exception and reconnected again, sometimes connecting was success. From what I saw, the exception happened when just connecting to server as well as connecting to server and waiting a long time without no binlog events. I also observed the exception came after immediately FORMAT_DESCRIPTION event was parsed. However, if reaching EOF of binlog is the condition of this exception, why it did not happen all the time?

Ability to deserialize TABLE_MAP and ROTATE events to byte arrays

Related to #28 since the library depends on those being deserialized to a specific type.

Address all the "todo"s

Failed to deserialize data

Stack Trace
Exception:
com.github.shyiko.mysql.binlog.event.deserialization.EventDataDeserializationException: Failed to deserialize data of EventHeaderV4{timestamp=1433181420000, eventType=UPDATE_ROWS, serverId=478847, headerLength=19, dataLength=183, nextPosition=38149950, flags=0}
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:140)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.nextEvent(EventDeserializer.java:116)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:501)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:328)
at com.github.shyiko.mysql.binlog.BinaryLogClient$4.run(BinaryLogClient.java:425)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at com.github.shyiko.mysql.binlog.io.BufferedSocketInputStream.read(BufferedSocketInputStream.java:51)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readWithinBlockBoundaries(ByteArrayInputStream.java:192)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:174)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readLong(ByteArrayInputStream.java:57)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeCell(AbstractRowsEventDataDeserializer.java:121)
at com.github.shyiko.mysql.binlog.event.deserialization.AbstractRowsEventDataDeserializer.deserializeRow(AbstractRowsEventDataDeserializer.java:96)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserializeRows(UpdateRowsEventDataDeserializer.java:72)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:58)
at com.github.shyiko.mysql.binlog.event.deserialization.UpdateRowsEventDataDeserializer.deserialize(UpdateRowsEventDataDeserializer.java:33)
at com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.deserializeEventData(EventDeserializer.java:134)
... 5 more

Using 0.1.1 version of mysql-binlog-connector-java

public void onCommunicationFailure(BinaryLogClient client, Exception ex) {
System.out.println("Communication failure ,Exception: "+ex.printStackTrace());
}

I'm getting above exception when any update/write/delete event occurs, saying not able to deserialize the event data.
As you suggested, I tried setting BinaryLogClient.setServerId(uniqueid) and also keepAliveInterval to 60 sec, but still exception persists

Use FormatDescriptionEvent to resolve checksum algorithm

EventDeserializer::deserializeEventData

How to handle binarylogclient connect during failovers

Hi
Locally Iam storing binlog details(name, position, serverid etc.,), to read binlogs from db cluster(i.e., master db).
If suppose any failover happens, and client connect to db cluster (but now to slave i.e., current master now), but the bin log details doesn't match with what we have locally. And when we try to connect with this, it get blocked in connect() method saying (connected to clusterip:port wit bin log name and position is bin.0000XX and YYYYYYY respectively). Please help me to handle the client connect when failover happens.

Thanks
Akram Syed

Make socket i/s buffering configurable

Issue connecting to master

Every time I try to connect to any master (mysql 5.5.6) I get the following error:

java.io.EOFException
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.read(ByteArrayInputStream.java:180)
at com.github.shyiko.mysql.binlog.io.ByteArrayInputStream.readZeroTerminatedString(ByteArrayInputStream.java:81)
at com.github.shyiko.mysql.binlog.network.protocol.GreetingPacket.(GreetingPacket.java:48)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:297)
at mem_search.Go.main(Go.java:23)

I've tried multiple database server and I tried remote connect as well as running it on the local host. Can't seem to get it to work. Any tips?

BinaryLogClientIntegrationTest has timezone-related failures

I'm in the America/Los_Angeles timezone. I ran into these two problems while running the integration test. First: the test setup fails:

Tests run: 38, Failures: 1, Errors: 0, Skipped: 33, Time elapsed: 1.235 sec <<< FAILURE! - in TestSuite
setUp(com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest)  Time elapsed: 0.593 sec  <<< FAILURE!
java.sql.SQLException: Unknown or incorrect time zone: '--7:00'
    at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2713)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2663)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:888)
    at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:730)
    at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection$1.execute(BinaryLogClientIntegrationTest.java:565)
    at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection$1.execute(BinaryLogClientIntegrationTest.java:556)
    at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection.execute(BinaryLogClientIntegrationTest.java:574)
    at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection.<init>(BinaryLogClientIntegrationTest.java:556)
    at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest$MySQLConnection.<init>(BinaryLogClientIntegrationTest.java:539)
    at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest.setUp(BinaryLogClientIntegrationTest.java:87)

The code to generate the timezone string for the SET time_zone command is wrong, and incorrectly generates two minus signs for those of us who live in the past (SET time_zone = '--7:00'). Commit 6eaa9a3 in my fork gets past this failure, but I'm not sure yet that this is the right fix (see below).

After applying that change, however, I do run into a different failure, seemingly related to timezones and daylight saving time:

Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.325 sec <<< FAILURE! - in TestSuite
testDeserializationOfDifferentColumnTypes(com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest)  Time elapsed: 0.777 sec  <<< FAILURE!
java.lang.AssertionError: Lists differ at element [0]: 1989-03-18 01:02:03.0 != 1989-03-18 00:02:03.0 expected [1989-03-18 01:02:03.0] but found [1989-03-18 00:02:03.0]
    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:494)
    at org.testng.Assert.assertEquals(Assert.java:123)
    at org.testng.Assert.assertEquals(Assert.java:549)
    at org.testng.Assert.assertEquals(Assert.java:668)
    at org.testng.Assert.assertEquals(Assert.java:725)
    at com.github.shyiko.mysql.binlog.BinaryLogClientIntegrationTest.testDeserializationOfDifferentColumnTypes(BinaryLogClientIntegrationTest.java:206)

If I modify the test case to '1989-05-18 01:02:03.0', then it passes. This likely has some connection to the SET time_zone = '-7:00', because on 1989-03-18 01:02:03.0, California was at a -8:00 offset, whereas today (2014-04-18) we're -7:00. So there's the chance as well that this test could variably succeed or fail depending on the season.

Why not support pull instead of push in BinaryLogClient ?

This may solve a lot of problem, like #15 ,for currently Listener, I have to use a BlockingQueue to implement some pull style code.

[feature request] expose methods for handling raw event data.

I would like to publish the raw event data, and then parse the event data on the subscribers to let them handle the events themselves.

Example with WriteRowsEventData

Hello,

I am playing around with mysql-binlog-connector-java and I am failing to understand how exactly do I loop and get the values from a WriteRowsEventData (or UpdatedRowsEventData).

Let's say we got the Event inside onEvent(Event event)...what's next?
In open-replicator you were doing row.getColumns() and you got the column values.

It's not very clear what to do after you got the rows and how to deserialize (or does this happen automatically?)

Cheers

Provide GTID EventDataDeserializer out-of-the-box

FK relationship

How to get notification for tables having fk realtionship like on delete cascade etc..

No field name in TABLE_MAP_EVENT?

Hi.

This is not a problem about 'mysql-binlog-connector-java'.

I just find out that there is no field name in TableMapEventData. how can i regenerate the SQL from WRITE_ROWS,UPDATE_ROWS,DELETE_ROWS.or there is other method to update slave mysql database?

Thanks for help.

how distribute processing mysql binlog byte data ?

distributed processing mysql binlog byte data but not deserializer it.
not deserializer bytes. only distributed .
sorrry for my bad english.
thank you.

How to get column names of a table?

In TableMapEventData I can get DB name, table name, column types etc. But there's no info about column names.

How to get column names of a table? Is there a feature in mysql-binlog-connector-java to do that? Or must I use other tools like JDBC? It's very helpful if there's instruction about this in the README.

(Sorry this is a question. I don't know a better place to post it so I post it here.)

Obtain the values for before and after

Any chance I can get some instructions about obtaining the row values before and after a modification in case of RBR ?

I couldn't find any documentation about this.

Replication lag

Hi,

First of all, mysql-binlog-connector-java is a really great tool. We use this to create a test environment by replicating the production database in a special way. The connector works great but we experienced an increasing lag behind the master server even without any event listeners. The connection between the servers are good and we do not have lag with mysql's master-slave replication.

Do you have any idea how to improve the processing time somehow? What do you think what causes the lag?

binlog_format question

Hi,

mysql-binlog-connector-java is a great tool. I have tried some other mysql replication tools, like Databus(LinkedIn), canal(alibaba).They all need to set binlog_format=ROW.

While using mysql-binlog-connector-java, I tried 'MIXED' and 'ROW'. Both can work.
I got 'WRITE_ROWS' event while using 'ROW' format. 'QUERY' event while using 'MIXED' format.It seems both format are supported.

Is it OK to use binlog_format='MIXED' and mysql-binlog-connector-java, in my production environment?

thanks.

Server ID

Hello,

I am looking at the BinaryLogClient and I realize that you can not set a server id but it fetches automatically the id of the master which I believe it's not the right behavior.

A unique server id must be set for master and all replication servers as you can also read here:
http://dev.mysql.com/doc/refman/5.0/en/replication-options-slave.html

Server ID.  On the master and each slave, you must use the server-id option to establish 
a unique replication ID in the range from 1 to 232 – 1. “Unique” means that each ID must 
be different from every other ID in use by any other replication master or slave.

Am I missing something?

Cheers

Provide helper class(es) for overcoming "Implementation notes"