nim4/dbshield

Database firewall written in Go

dbshield's Issues

DBShield run into i/o timeout when receiving response from mysql 5.7 as backend server

When test against MySQL 5.7 as backend server, DBShield failed into i/o timeout when read packets fromserver. But the server did send back packets to DBShield. But this doesn't recurrent for Mysql 5.5 or 5.6. I am not quit sure about it. Please help.

debug log:

2017/10/19 17:18:13 [INFO]  Config file: /path/to/dbshield.yml
2017/10/19 17:18:13 [INFO]  Listening: 0.0.0.0:13307
2017/10/19 17:18:13 [INFO]  Backend: mysql (10.10.10.57:3306)
2017/10/19 17:18:13 [INFO]  Protect: false
2017/10/19 17:18:13 [INFO]  Web interface on https://127.0.0.1:8070/
2017/10/19 17:18:20 [DEBUG] Connected from: 127.0.0.1:56552
2017/10/19 17:18:20 [DEBUG] Connected to: 10.10.10.57:3306
2017/10/19 17:18:20 [DEBUG] SSL connection
2017/10/19 17:18:20 [DEBUG] Client handshake done
2017/10/19 17:18:20 [DEBUG] Server handshake done
2017/10/19 17:18:20 [DEBUG] Username: root
2017/10/19 17:18:20 [DEBUG] Database: mysql_native_password
2017/10/19 17:18:20 [DEBUG] SSL bit: true
2017/10/19 17:18:20 [DEBUG] Query: show databases
2017/10/19 17:18:20 [DEBUG] action: learning
2017/10/19 17:18:20 [WARN]  data recevied: ���K��def�information_schemSCHEMATSCHEMATDatabase
                                                                                            SCHEMA_NAME
                                                                                !���information_schema��mysql��performance_schema���sys"
2017/10/19 17:18:30 [WARN]  read tcp 10.2.1.5:56553->10.10.10.57:3306: i/o timeout
2017/10/19 17:18:54 [DEBUG] Connected from: 127.0.0.1:56593
2017/10/19 17:18:54 [DEBUG] Connected to: 10.10.10.57:3306
2017/10/19 17:18:54 [DEBUG] SSL connection
2017/10/19 17:18:54 [DEBUG] Client handshake done
2017/10/19 17:18:54 [DEBUG] Server handshake done
2017/10/19 17:18:54 [DEBUG] Username: root
2017/10/19 17:18:54 [DEBUG] Database: mysql_native_password
2017/10/19 17:18:54 [DEBUG] SSL bit: true
2017/10/19 17:18:54 [DEBUG] Query: select * from first where name<1719
2017/10/19 17:18:54 [DEBUG] action: learning
2017/10/19 17:18:54 [WARN]  data recevied: �#3D000No database selected
2017/10/19 17:19:04 [WARN]  read tcp 10.2.1.5:56594->10.10.10.57:3306: i/o timeout

env

DBShield Version: 1.0.0-beta4

$ go version
go version go1.9.1 darwin/amd64

$ mysql --version
mysql  Ver 14.14 Distrib 5.7.19, for Linux (x86_64) using  EditLine wrapper

Will the project continue to be updated?

hi，
Will the project continue to be updated?

[WARN] runtime error: slice bounds out of range

Hi nim4,

I found this issue on DBShield when my website query database on port 5000 that DBShield listening.
Do you know how to fix it or i configured something wrong ?

go run main.go

2016/12/29 09:45:41 [INFO] Config file: /etc/dbshield.yml
2016/12/29 09:45:41 [INFO] Listening: 0.0.0.0:5000
2016/12/29 09:45:41 [INFO] Backend: mysql (127.0.0.1:3306)
2016/12/29 09:45:41 [INFO] Protect: false
2016/12/29 09:45:41 [INFO] Web interface on https://XXX.XXX.XXX.XXX:8070/
2016/12/29 09:46:14 [WARN] runtime error: slice bounds out of range
2016/12/29 09:52:07 [WARN] runtime error: slice bounds out of range
2016/12/29 09:52:11 [WARN] runtime error: slice bounds out of range
2016/12/29 09:52:13 [WARN] runtime error: slice bounds out of range

/etc/dbshield.yml

##############################
### General               ###
#############################

# Operating mode
# possible values:
#       learning - add queries to training set (default)
#       protect - take action against abnormal queries
mode: learning

# DBMS type(default: mysql)
dbms: mysql

# IP address to listen on(default: 0.0.0.0)
listenIP: 0.0.0.0

# Port number to listen on (0 means dbms default port ex. 3306 for mysql)
listenPort: 5000

# IP address of the server
targetIP: 127.0.0.1

# Port number of the dbms (0 means dbms default port ex. 3306 for mysql)
targetPort: 3306

# TLS
tlsPrivateKey:  cert/server-key.pem
tlsCertificate: cert/server-cert.pem

# Directory to save internal database(default: in model directory under OS's temp directory)
dbDir: "/tmp/model/"

# Databasse synchronisation interval, 0 will force sync after each commit.
# Valid units are "ns", "us" (or "µs"), "ms", "s", "m" and "h". (default: 5s)
SyncInterval: 5s

#############################
### HTTP Service          ###
#############################
# Run http interface(default: yes)
http: yes

# Serve https (default: yes)
httpSSL: yes

# IP address to listen on(default: 127.0.0.1)
#httpIP: 127.0.0.1
httpIP: XXX.XXX.XXX.XXX

# Port number to listen on(default: 8070)
httpPort: 8070

# Password
httpPassword: admin01

#############################
### Protect               ###
#############################

# Action to take against abnormal requests
# possible values:
#       drop  - close the connection (default)
#       pass  - pass the query to server
action: drop

# Properties to check for abnormality
#
# possible values: (you can choose more than one - comma separated)
#       user     - username
#       source   - request sourece address
#
# Note: leave empty to avoid adiditional checks
additionalChecks:  user,source

#############################
### Logging               ###
#############################

# Log depth
#       1   warning
#       2   info
#       4   debug
# default: 3 (warning + info)
logLevel: 3

# Log path
# possible values:
#       /path/to/file
#       stdout
#       stderr (default)
logPath: stderr

Drop action cause DBShield existing from command line

When using protect mode from command line, DBShield existed with errors. I think it may need to drop the connection without make itself existed.

DBShield version: commit b07fb6e
Go version: go version go1.9 darwin/amd64

> $GOPATH/bin/DBShield -c /etc/dbshield.yml

....

2017/09/21 18:52:14 [DEBUG] Query: select * from first where iii='中文'
2017/09/21 18:52:14 [WARN]  Invalid pattern
panic: Dropping connection

goroutine 10 [running]:
github.com/nim4/DBShield/dbshield/utils.ActionDrop(0xc42013802d, 0x26)
        /Users/qiwihui/Development/gopath/src/github.com/nim4/DBShield/dbshield/utils/action.go:5 +0x39
github.com/nim4/DBShield/dbshield/dbms.processContext(0xc42013802d, 0x26, 0x3b, 0xc420078324, 0x4, 0x39, 0xc42001438c, 0x10, 0x10, 0xc42007833e, ...)
        /Users/qiwihui/Development/gopath/src/github.com/nim4/DBShield/dbshield/dbms/utils.go:321 +0x18c
github.com/nim4/DBShield/dbshield/dbms.(*MySQL).Handler(0xc4200ba240, 0x0, 0x0)
        /Users/qiwihui/Development/gopath/src/github.com/nim4/DBShield/dbshield/dbms/mysql.go:91 +0x528
github.com/nim4/DBShield/dbshield.handleClient(0x1865d00, 0xc42000e568, 0xc420118090, 0xc420124008, 0x0)
        /Users/qiwihui/Development/gopath/src/github.com/nim4/DBShield/dbshield/utils.go:161 +0x350
created by github.com/nim4/DBShield/dbshield.mainListner
        /Users/qiwihui/Development/gopath/src/github.com/nim4/DBShield/dbshield/dbshield.go:162 +0x38a

sql statement: select * from first where iii='中文'; where there is no iii in my first table

DBShield run into i/o timeout when mysql return data reaches a certain size

When try to handle return data from mysql, dbshield run into i/o timeout.

reproducing

1. In server A, create mysql db dbsheild and table first with

create table first (name varchar(50));

and initial data with

for i in {0..5000}; do mysql -h 127.0.0.1 -P3306 -uroot -ppassword dbshield -Bse "insert into first (name) values (${i})"; done

2. In server B, query data without DBShield, and it works fine even when I change 2,000 to 5,000:

mysql -h server-A-ip -P3306 -uroot -ppassword dbshield -Bse "select * from first where name<2000;"

3. In server B, setting up dbsheild and with default config from another server and modify follow config:

listenIP: 0.0.0.0
listenPort: 13308
targetIP: <server-A-ip>
targetPort: 3306
logLevel: 7

then it reprocuded.

$ mysql -h 127.0.0.1 -P13308 -uroot -ppassword dbshield -Bse "select * from first where name<1000;"

<return with expected data>

$ mysql -h 127.0.0.1 -P13308 -uroot -ppassword dbshield -Bse "select * from first where name<1716;"

<return with expected data>

$ mysql -h 127.0.0.1 -P13308 -uroot -ppassword dbshield -Bse "select * from first where name<1717;"

ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query

here is the debug log:

# ~/go/bin/DBShield -c ~/dbshield/dbshield.yml 
2017/09/18 09:19:59 [INFO]  Config file: /root/dbshield/dbshield.yml
2017/09/18 09:19:59 [INFO]  Listening: 0.0.0.0:13308
2017/09/18 09:19:59 [INFO]  Backend: mysql (<server-A-ip>:3306)
2017/09/18 09:19:59 [INFO]  Protect: false
2017/09/18 09:19:59 [INFO]  Web interface on https://127.0.0.1:8070/
2017/09/18 09:20:38 [DEBUG] Connected from: 127.0.0.1:38886
2017/09/18 09:20:38 [DEBUG] Connected to: <server-A-ip>:3306
2017/09/18 09:20:38 [DEBUG] Username: root
2017/09/18 09:20:38 [DEBUG] Database: dbshield
2017/09/18 09:20:38 [DEBUG] SSL bit: false
2017/09/18 09:20:38 [DEBUG] Query: select @@version_comment limit 1
2017/09/18 09:20:38 [DEBUG] Query: select * from first where name<1000
2017/09/18 09:20:56 [DEBUG] Connected from: 127.0.0.1:38892
2017/09/18 09:20:56 [DEBUG] Connected to: <server-A-ip>:3306
2017/09/18 09:20:56 [DEBUG] Username: root
2017/09/18 09:20:56 [DEBUG] Database: dbshield
2017/09/18 09:20:56 [DEBUG] SSL bit: false
2017/09/18 09:20:56 [DEBUG] Query: select @@version_comment limit 1
2017/09/18 09:20:56 [DEBUG] Query: select * from first where name<1717
2017/09/18 09:21:01 [WARN]  read tcp 127.0.0.1:13308->127.0.0.1:38892: i/o timeout

env

DBShield Version: 1.0.0-beta4

$ go version
go version go1.6.3 linux/amd64

$ uname -a
Linux qiwihui 4.10.8-041008-generic #201703310531 SMP Fri Mar 31 09:33:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ mysql --version 
mysql  Ver 14.14 Distrib 5.5.57, for debian-linux-gnu (x86_64) using readline 6.3

Web Interface 404

DBShield -c dbshield.yml

404 page not found

My Config

##############################
### General               ###
#############################

# Operating mode
# possible values:
#       learning - add queries to training set (default)
#       protect - take action against abnormal queries
mode: learning

# DBMS type(default: mysql)
dbms: mariadb

# IP address to listen on(default: 0.0.0.0)
listenIP: 0.0.0.0

# Port number to listen on (0 means dbms default port ex. 3306 for mysql)
listenPort: 5000

# IP address of the server
targetIP: 127.0.0.1

# Port number of the dbms (0 means dbms default port ex. 3306 for mysql)
targetPort: 0

# TLS
tlsPrivateKey:  /etc/ssl/maxdb_key.pem
tlsCertificate: /etc/ssl/maxdb_certificate.pem

# Directory to save internal database(default: in model directory under OS's temp directory)
dbDir: "/tmp/model/"

# Connection timeout
# Valid units are "ns", "us" (or "µs"), "ms", "s", "m" and "h". (default: 5s)
timeout: 5s

# Databasse synchronisation interval, 0 will force sync after each commit.
# Valid units are "ns", "us" (or "µs"), "ms", "s", "m" and "h". (default: 5s)
SyncInterval: 5s

#############################
### HTTP Service          ###
#############################
# Run http interface(default: yes)
http: yes

# Serve https (default: yes)
httpSSL: no

# IP address to listen on(default: 127.0.0.1)
httpIP: 127.0.0.1

# Port number to listen on(default: 8070)
httpPort: 8070

# Password
httpPassword: ChangeMe

#############################
### Protect               ###
#############################

# Action to take against abnormal requests
# possible values:
#       drop  - close the connection (default)
#       pass  - pass the query to server
action: drop

# Properties to check for abnormality
#
# possible values: (you can choose more than one - comma separated)
#       user     - username
#       source   - request sourece address
#
# Note: leave empty to avoid adiditional checks
additionalChecks:  user,source

#############################
### Logging               ###
#############################

# Log depth
#       1   warning
#       2   info
#       4   debug
# default: 3 (warning + info)
logLevel: 3

# Log path
# possible values:
#       /path/to/file
#       stdout
#       stderr (default)
logPath: stderr

Was expecting something similar to the screenshot.

DBShield Version: 1.0.0-beta4
Go Version: go1.6.2 linux/amd64
OS Version: Ubuntu 16.04.2 LTS Xenial

What Oracle versions are supported

What Oracle versions are supported