I faced 2 more instances of MySQL crash, I deployed the plugin on one of our Replica DB, but it crashed twice in span of 8 hours on its own.
mysql> status
mysql Ver 14.14 Distrib 5.5.12, for Linux (x86_64) using readline 5.1
Connection id: 16215
Current database:
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 5.5.12-log MySQL Community Server (GPL)
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: utf8
Db characterset: utf8
Client characterset: utf8
Conn. characterset: utf8
UNIX socket: /var/lib/mysql/mysql.sock
Uptime: 1 day 28 min 35 sec
Threads: 74 Questions: 20112136 Slow queries: 210 Opens: 11200 Flush tables: 1 Open tables: 1000 Queries per second avg: 228.248
mysql> show global status like 'AUDIT_version';
+---------------+-------------+
| Variable_name | Value |
+---------------+-------------+
| AUDIT_version | 1.0.0-99999 |
+---------------+-------------+
1 row in set (0.00 sec)
Audit Plugin parameters in my.cnf(INSTANCE 1)
plugin-load=AUDIT=libaudit_plugin.so
audit_json_file=ON
audit_json_log_file=/var/lib/mysql/mysql-audit.log
audit_json_socket=OFF
audit_validate_checksum=OFF
audit_record_cmds=select,Connect,Quit
audit_record_objs=ebay.,ebay_smpi.,shipping.,dwh.,analytics.*,{}
ERROR LOG (INSTANCE 1)
130109 5:12:05 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=1073741824
read_buffer_size=524288
max_used_connections=52
max_threads=2000
thread_count=41
connection_count=41
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4143341 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x25cec360
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fecbeb9de50 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x796609]
/usr/sbin/mysqld(handle_segfault+0x380)[0x4fcdf0]
/lib64/libpthread.so.0(+0xf4a0)[0x7ff2658834a0]
/lib64/libc.so.6(+0x130752)[0x7ff264b3b752]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0xca7a)[0x7ff216dc9a7a]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0xd078)[0x7ff216dca078]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x205)[0x5784c5]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x337)[0x60d3b7]
/usr/sbin/mysqld(handle_one_connection+0x54)[0x60d424]
/lib64/libpthread.so.0(+0x77f1)[0x7ff26587b7f1]
/lib64/libc.so.6(clone+0x6d)[0x7ff264af092d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query ((nil)): is an invalid pointer
Connection ID (thread ID): 8193
Status: NOT_KILLED
Session ID 8193 details (INSTANCE 1)
{"msg-type":"activity","date":"1357688525201","thread-id":"8193","query-id":"0","user":"username","priv_user":"username","ip":"Some IP","cmd":"Connect","query":"Connect"}
{"msg-type":"activity","date":"1357688525203","thread-id":"8193","query-id":"15158714","user":"username","priv_user":"username","ip":"Some IP","cmd":"select","query":"select @@version_comment limit 1"}
{"msg-type":"activity","date":"1357688525204","thread-id":"8193","query-id":"15158717","user":"username","priv_user":"username","ip":"Some IP","cmd":"select","objects":[{"db":"dwh","name":"table_name","obj_type":"TABLE"}],"query":"SELECT CONCAT("total_line_read : ",lines_input," total_lines_written: ",lines_output," lines_rejected: ",lines_input - lines_output -lines_rejected," lines_error: ",lines_rejected ," flow_completed_on: ",logdate) Message_below FROM dwh.table_name WHERE columnname = 'xxx_xxxxxx_xxxx_xxxxxxxx' ORDER BY logdate DESC LIMIT 0 , 1"}
Audit Plugin parameters in my.cnf(INSTANCE 2)
plugin-load=AUDIT=libaudit_plugin.so
audit_json_file=OFF
audit_json_log_file=/var/lib/mysql/mysql-audit.log
audit_json_socket=OFF
audit_validate_checksum=OFF
audit_record_cmds=select,Connect,Quit
audit_record_objs=ebay.,ebay_smpi.,shipping.,dwh.,analytics.*,{}
ERROR LOG (INSTANCE 2)
130109 12:38:43 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=1073741824
read_buffer_size=524288
max_used_connections=53
max_threads=2000
thread_count=50
connection_count=50
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4143341 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x24d65430
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f4a5183de50 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x796609]
/usr/sbin/mysqld(handle_segfault+0x380)[0x4fcdf0]
/lib64/libpthread.so.0(+0xf4a0)[0x7f4ff8f2e4a0]
/lib64/libc.so.6(+0x130752)[0x7f4ff81e6752]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0xca7a)[0x7f4faadc9a7a]
/usr/lib64/mysql/plugin/libaudit_plugin.so(+0xd078)[0x7f4faadca078]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x205)[0x5784c5]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x337)[0x60d3b7]
/usr/sbin/mysqld(handle_one_connection+0x54)[0x60d424]
/lib64/libpthread.so.0(+0x77f1)[0x7f4ff8f267f1]
/lib64/libc.so.6(clone+0x6d)[0x7f4ff819b92d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query ((nil)): is an invalid pointer
Connection ID (thread ID): 5133
Status: NOT_KILLED
Session ID 5133 details (INSTANCE 1)
No Session details available as audit_json_file=OFF