xiaomi/gaea

Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.

gaea's Issues

Invalid global sequence id

1. 生成的全局自增id不连续， 导致以该列作为分片分表键时，分片不均匀

2. 重新开启一个客户端后，会生成重复id

规则

{
    "name": "test_namespace_1",
    "online": true,
    "read_only": false,
    "allowed_dbs": {
        "gaea_test": true
    },
    "slow_sql_time": "1000",
    "black_sql": [
        ""
    ],
    "allowed_ip": null,
    "slices": [
        {
            "name": "slice-0",
            "user_name": "root",
            "password": "root",
            "master": "127.0.0.1:3306",
            "slaves": null,
            "statistic_slaves": null,
            "capacity": 12,
            "max_capacity": 24,
            "idle_timeout": 60
        },
       {
            "name": "slice-1",
            "user_name": "root",
            "password": "root",
            "master": "127.0.0.1:3306",
            "slaves": null,
            "statistic_slaves": null,
            "capacity": 12,
            "max_capacity": 24,
            "idle_timeout": 60
        }
    ],
    "shard_rules": [
        {
            "db": "gaea_test",
            "table": "sequence",
            "type": "mod",
            "key": "id",
            "locations": [
                1,
                1
            ],
            "slices": [
                "slice-0",
                "slice-1"
            ]
        }
    ],
    "users": [
        {
            "user_name": "root",
            "password": "root",
            "namespace": "test_namespace_1",
            "rw_flag": 2,
            "rw_split": 1,
            "other_property": 0
        }
    ],
    "default_slice": "slice-0",
    "global_sequences":[
        {
            "db": "gaea_test",
            "table": "sequence",
            "type": "mycat",
            "pk_name": "id",
            "slice_name": "slice-0"
        }
    ]
}

请问后期还更新吗

如题：请问后期还更新吗

什么时候能支持mysql8客户端？

mysql8客户端访问：

1. 密码验证阶段会失败。
2. utf8mb4还缺少不少collation配置

gaea-cc在配置两阶段提交时，没考虑proxy返回的错误信息，可能会导致proxy进程崩溃。

gaea-cc在配置两阶段提交时，请求 /api/proxy/config/prepare/:namespace 接口后，没有判断接口返回的内容，只判断了请求是否成功，接着直接去请求了/api/proxy/config/commit/:namespace接口，在以下情况会导致proxy进程崩溃。

如提交一个不存在的分片类型，prepare接口会报 “err: unknown rule type”，但是cc忽略了这个错误直接去请求 commit接口后，proxy通过ReloadNamespaceCommit方法把 Manager.switchIndex置为1，此时 Manager.namespaces下标1为nil（因为prepare接口并没有解析成功刚刚提交的错误分片类型namespace配置），导致Manager.startConnectPoolMetricsTask()方法中的这行代码for nameSpaceName, _ := range m.namespaces[current].namespaces 引发错误"invalid memory address or nil pointer dereference"。

gaea-proxy panic when commit config without prepare

IsRWSplit判断错误

Gaea/proxy/server/namespace.go

Line 216 in cf47722

return n.userProperties[user].RWFlag == models.ReadWriteSplit

这里应该是

return n.userProperties[user].RWSplit == models.ReadWriteSplit

DESCRIBE table 这种语法不支持

DESCRIBE table 这种语法不支持,执行后返回 SHOW COLUMNS IN table

jetbrain DataGrip connect error

Connection to @xxx.xxx.xx.xx failed.
[08001] Could not create connection to database server. Attempted reconnect 3 times. Giving up.
使用mysql自带客户端和Navicat连接没问题，经过测试发现使用jdbc mysql-connector 5.1的驱动可以，最新的8.0不行

请问如何将namespace的json配置数据导入到etcd呢？

对etcd不是很熟悉，用这样的命令将etc/namespaces下的json串导入到etcd后

./etcdctl put /gaea/file/namespace/test_namespace_1.json "{"name":"test_namespace_1","online":true,"read_only":false,"allowed_dbs":{"test_db":true},"slow_sql_time":"1000","black_sql":[""],"allowed_ip":null,"slices":[{"name":"slice-0","user_name":"root","password":null,"master":"127.0.0.1:3306","slaves":null,"statistic_slaves":null,"capacity":12,"max_capacity":24,"idle_timeout":60}],"shard_rules":[{"db":"test_db","table":"log","type":"date_month","key":"created_at","slices":["slice-0"],"date_range":["201804-201910"]}],"users":[{"user_name":"root","password":"root","namespace":"test_namespace_1","rw_flag":2,"rw_split":1,"other_property":0}],"default_slice":"slice-0","global_sequences":null}"

启动Gaea服务

bogon:Gaea user$ ./bin/gaea -config etc/gaea.ini
Build Version Information:Version: 666d59e9f7ea563733299d97260ee43307881d70
GitRevision: 666d59e9f7ea563733299d97260ee43307881d70
User: [email protected]
GolangVersion: go1.13
BuildStatus: Clean
BuildTime: 2019-09-16--14:08:20

服务无法启动，也没有相关的报错信息，请问应该如何正确的把文本模式下正常工具的namespace配置导入etcd呢？

user配置具体属性

如下图，这些配置的具体属性代表什么

win系统可以用么

win系统可以用么

能提供下grafana的监控模版吗

能提供下grafana的监控模版吗

应用报大量 “resource pool timed out”，大佬帮分析下原因。

昨天应用报大量“SQLSTATE[HY000]: General error: 1105 unknown error: resource pool timed out (SQL: xxxxxxxx”错误，我查看Gaea源码，发现 resource pool timed out 错误在 util\resource_pool.go的第36行定义。

在这之前的一小段时间应用大量报" SQLSTATE[HY000]: General error: 1105 unknown error: Conn 0: Write(header) failed: write tcp 10.5.1.128:53996->10.0.8.48:3306: write: connection reset by peer (SQL:xxxxxxxxx"错误，在mysql/conn.go的402行发现 Write(header) failed:定义；与此同时Gaea日志中出现几十条"[shazam_proxy] [nicetuan-middle-payment-prod-002] [NOTICE] [900000001] [main.main.func1:main.go:90] Ignore broken pipe signal"。

重启proxy后，应用恢复正常，而且这些日志只出现在一台应用服务器上（我把应用和proxy部署在同一台机器上），其它7台机器没事。
没弄明白这个问题出现的原因，麻烦大佬帮忙分析下。

use -ldflags to pass through version and compiler infomation during compile time

in core/version.go, you guys defined version and compiler info with hard code.

maybe we can use -ldflags to pass this information during compile time?

support encrypt user name and password in namespace config

[mysql]:unknown error: resource pool timed out[sql]

[mysql]:unknown error: resource pool timed out[sql]

[Feature Request] - strict update mode

hi,team:

I am trying to design a feature called strict update mode. it's meaning just allow execute UPDATE SQL with the primary key or which configured fields. the proxy will reject the update SQL when can't match configure.

Samples:

configure:

table1: just allow update with the primary key.
table2: just allow update when included user_id or id on where clause.

examples:

update table1 field_1 = 1 where id = 1. [pass]
update table1 field_1 = 1 where field_2 = 1. [reject]
update table2 field_1 = 1 where id = 1. [pass]
update table2 field_1 = 1 where user_id = 1 [pass]
update table2 field_1 = 1 where field_2 = 1. [reject].

how about the idea?

prepare statements missing type info will cause panic

关于rw_flag和rw_split

读写标识rw_flag的值有哪几个呢？取值分别表示什么意思？
读写分离rw_split的取值同样存在疑问

make时报错

go: finding gopkg.in/resty.v1 v1.12.0
go: google.golang.org/[email protected]: unrecognized import path "google.golang.org/genproto" (https fetch: Get https://google.golang.org/genproto?go-get=1: dial tcp 216.239.37.1:443: i/o timeout)
go: error loading module requirements
Makefile:12: recipe for target 'gaea' failed
make: *** [gaea] Error 1

gaea-agent、gaea-web

gaea-agent、gaea-web 这两个模块开源吗

make编译失败

编译代码为master上现拉的代码，具体的报错信息如下所示：

编译信息：
[root@ip-10-0-21-222 Gaea]# make
cd parser && make && cd ..
make[1]: 进入目录“/data/Gaea/parser”
make parser
make[2]: 进入目录“/data/Gaea/parser”
./goyacc -o /dev/null parser.y
Parse table entries: 946812 of 2383712, x 16 bits == 1893624 bytes
./goyacc -o parser.go parser.y 2>&1 | egrep "(shift|reduce)/reduce" | awk '{print} END {if (NR > 0) {print "Find conflict in parser.y. Please check y.output for more information."; exit 1;}}'
rm -f y.output
make[2]: 离开目录“/data/Gaea/parser”
#@echo "gofmt (simplify)"
#@ gofmt -s -l -w . 2>&1 | awk '{print} END{if(NR>0) {exit 1}}'
make[1]: 离开目录“/data/Gaea/parser”
go build -o /data/Gaea/bin/gaea -pkgdir=/root/go/pkg/linux_amd64 -ldflags "-extldflags -static -X github.com/XiaoMi/Gaea/core.buildVersion=666d59e9f7ea563733299d97260ee43307881d70-dirty -X github.com/XiaoMi/Gaea/core.buildGitRevision=666d59e9f7ea563733299d97260ee43307881d70-dirty -X github.com/XiaoMi/Gaea/core.buildUser=root -X github.com/XiaoMi/Gaea/core.buildHost=ip-10-0-21-222.ap-east-1.compute.internal -X github.com/XiaoMi/Gaea/core.buildStatus=Modified -X github.com/XiaoMi/Gaea/core.buildTime=2019-09-16--08:21:47" github.com/XiaoMi/Gaea/cmd/gaea
go: downloading github.com/gin-contrib/gzip v0.0.1
go: downloading github.com/gin-gonic/gin v1.3.0
go: downloading github.com/go-ini/ini v1.42.0
go: downloading github.com/pingcap/tipb v0.0.0-20190226124958-833c2ffd2fe7
go: downloading github.com/emirpasic/gods v1.12.0
go: downloading github.com/pingcap/errors v0.11.1
go: downloading github.com/coreos/etcd v3.3.13+incompatible
go: downloading github.com/prometheus/client_golang v0.9.2
go: extracting github.com/gin-contrib/gzip v0.0.1
go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548
go: extracting github.com/pingcap/errors v0.11.1
go: extracting github.com/emirpasic/gods v1.12.0
go: extracting github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548
go: downloading github.com/remyoudompheng/bigfft v0.0.0-20190321074620-2f0d2b0e0001
go: extracting github.com/prometheus/client_golang v0.9.2
go: downloading github.com/prometheus/common v0.0.0-20181126121408-4724e9255275
go: downloading github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: extracting github.com/go-ini/ini v1.42.0
go: extracting github.com/pingcap/tipb v0.0.0-20190226124958-833c2ffd2fe7
go: downloading github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: downloading github.com/golang/protobuf v1.2.0
go: downloading github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
go: extracting github.com/gin-gonic/gin v1.3.0
go: downloading github.com/mattn/go-isatty v0.0.7
go: downloading gopkg.in/yaml.v2 v2.2.1
go: extracting github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: downloading github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f
go: extracting github.com/remyoudompheng/bigfft v0.0.0-20190321074620-2f0d2b0e0001
go: downloading gopkg.in/go-playground/validator.v8 v8.18.2
go: extracting github.com/prometheus/common v0.0.0-20181126121408-4724e9255275
go: extracting github.com/mattn/go-isatty v0.0.7
go: downloading golang.org/x/sys v0.0.0-20190422165155-953cdadca894
go: extracting github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: downloading github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3
go: extracting gopkg.in/go-playground/validator.v8 v8.18.2
go: extracting gopkg.in/yaml.v2 v2.2.1
go: extracting github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
go: extracting github.com/golang/protobuf v1.2.0
go: extracting github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3
go: extracting github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: extracting github.com/matttproud/golang_protobuf_extensions v1.0.1
go: extracting github.com/coreos/etcd v3.3.13+incompatible
go: extracting golang.org/x/sys v0.0.0-20190422165155-953cdadca894
go: downloading github.com/modern-go/reflect2 v1.0.1
go: downloading github.com/json-iterator/go v1.1.6
go: downloading github.com/coreos/go-semver v0.2.0
go: extracting github.com/coreos/go-semver v0.2.0
go: extracting github.com/modern-go/reflect2 v1.0.1
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: extracting github.com/json-iterator/go v1.1.6
go: extracting github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: finding github.com/go-ini/ini v1.42.0
go: finding github.com/coreos/etcd v3.3.13+incompatible
go: finding github.com/json-iterator/go v1.1.6
go: finding github.com/pingcap/errors v0.11.1
go: finding github.com/coreos/go-semver v0.2.0
go: finding github.com/modern-go/reflect2 v1.0.1
go: finding github.com/prometheus/client_golang v0.9.2
go: finding github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: finding github.com/gin-contrib/gzip v0.0.1
go: finding github.com/gin-gonic/gin v1.3.0
go: finding github.com/emirpasic/gods v1.12.0
go: finding github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548
go: finding github.com/pingcap/tipb v0.0.0-20190226124958-833c2ffd2fe7
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding github.com/remyoudompheng/bigfft v0.0.0-20190321074620-2f0d2b0e0001
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding github.com/golang/protobuf v1.2.0
go: finding github.com/prometheus/common v0.0.0-20181126121408-4724e9255275
go: finding github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/mattn/go-isatty v0.0.7
go: finding github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
go: finding golang.org/x/sys v0.0.0-20190422165155-953cdadca894
go: finding github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f
go: finding gopkg.in/go-playground/validator.v8 v8.18.2
go: finding gopkg.in/yaml.v2 v2.2.1
go build -o /data/Gaea/bin/gaea-cc -pkgdir=/root/go/pkg/linux_amd64 -ldflags "-extldflags -static -X github.com/XiaoMi/Gaea/core.buildVersion=666d59e9f7ea563733299d97260ee43307881d70-dirty -X github.com/XiaoMi/Gaea/core.buildGitRevision=666d59e9f7ea563733299d97260ee43307881d70-dirty -X github.com/XiaoMi/Gaea/core.buildUser=root -X github.com/XiaoMi/Gaea/core.buildHost=ip-10-0-21-222.ap-east-1.compute.internal -X github.com/XiaoMi/Gaea/core.buildStatus=Modified -X github.com/XiaoMi/Gaea/core.buildTime=2019-09-16--08:22:30" github.com/XiaoMi/Gaea/cmd/gaea-cc
go test -coverprofile=.coverage.out ./...
go: downloading github.com/pingcap/check v0.0.0-20190102082844-67f458068fc8
go: extracting github.com/pingcap/check v0.0.0-20190102082844-67f458068fc8
go: finding github.com/pingcap/check v0.0.0-20190102082844-67f458068fc8
ok github.com/XiaoMi/Gaea/backend 0.021s coverage: 3.4% of statements
? github.com/XiaoMi/Gaea/cc [no test files]
? github.com/XiaoMi/Gaea/cc/proxy [no test files]
? github.com/XiaoMi/Gaea/cc/service [no test files]
ok github.com/XiaoMi/Gaea/cmd/gaea 0.006s coverage: 14.0% of statements
? github.com/XiaoMi/Gaea/cmd/gaea-cc [no test files]
? github.com/XiaoMi/Gaea/core [no test files]
? github.com/XiaoMi/Gaea/core/errors [no test files]
? github.com/XiaoMi/Gaea/log [no test files]
? github.com/XiaoMi/Gaea/log/xlog [no test files]
ok github.com/XiaoMi/Gaea/models 0.004s coverage: 22.2% of statements
ok github.com/XiaoMi/Gaea/models/etcd 0.002s coverage: 4.4% of statements
? github.com/XiaoMi/Gaea/models/file [no test files]
ok github.com/XiaoMi/Gaea/mysql 0.011s coverage: 34.3% of statements
ok github.com/XiaoMi/Gaea/parser 0.404s coverage: 84.1% of statements
ok github.com/XiaoMi/Gaea/parser/ast 0.149s coverage: 57.1% of statements
ok github.com/XiaoMi/Gaea/parser/auth 0.027s coverage: 0.0% of statements [no tests to run]
ok github.com/XiaoMi/Gaea/parser/format 0.029s coverage: 81.3% of statements
ok github.com/XiaoMi/Gaea/parser/model 0.025s coverage: 53.6% of statements
ok github.com/XiaoMi/Gaea/parser/opcode 0.002s coverage: 55.6% of statements
? github.com/XiaoMi/Gaea/parser/stmtctx [no test files]
ok github.com/XiaoMi/Gaea/parser/terror 0.012s coverage: 75.9% of statements
ok github.com/XiaoMi/Gaea/parser/tidb-types 0.031s coverage: 67.3% of statements
ok github.com/XiaoMi/Gaea/parser/tidb-types/json 0.034s coverage: 81.4% of statements
ok github.com/XiaoMi/Gaea/parser/tidb-types/parser_driver 0.014s coverage: 24.7% of statements
? github.com/XiaoMi/Gaea/parser/types [no test files]
--- FAIL: TestSelectKingshardDateYear (0.01s)
--- FAIL: TestSelectKingshardDateYear/select_from_tbl_ks_year_where_create_time<=1388505600 (0.00s)
plan_test.go:110: not equal, expect: map[slice-0:map[db_ks:[SELECT * FROM tbl_ks_year_2014 WHERE create_time<=1388505600]]], actual: map[]
--- FAIL: TestSelectKingshardDateYear/selectfrom_tbl_ks_year_where_create_time<=1388505600#01 (0.00s)
plan_test.go:110: not equal, expect: map[slice-0:map[db_ks:[SELECT * FROM tbl_ks_year_2014 WHERE create_time<=1388505600]]], actual: map[]
--- FAIL: TestSelectKingshardDateYear/selectfrom_tbl_ks_year_where_create_time>=1514736000 (0.00s)
plan_test.go:110: not equal, expect: map[slice-1:map[db_ks:[SELECT * FROM tbl_ks_year_2018 WHERE create_time>=1514736000 SELECT * FROM tbl_ks_year_2019 WHERE create_time>=1514736000]]], actual: map[slice-0:map[db_ks:[SELECT * FROM tbl_ks_year_2017 WHERE create_time>=1514736000]] slice-1:map[db_ks:[SELECT * FROM tbl_ks_year_2018 WHERE create_time>=1514736000 SELECT * FROM tbl_ks_year_2019 WHERE create_time>=1514736000]]]
--- FAIL: TestSelectKingshardDateYear/selectfrom_tbl_ks_year_where_create_time>=_1577808000 (0.00s)
plan_test.go:110: not equal, expect: map[], actual: map[slice-1:map[db_ks:[SELECT * FROM tbl_ks_year_2019 WHERE create_time>=1577808000]]]
panic: runtime error: index out of range [1] with length 1 [recovered]
panic: runtime error: index out of range [1] with length 1

goroutine 377 [running]:
testing.tRunner.func1(0xc000490600)
/usr/local/go/src/testing/testing.go:874 +0x3a3
panic(0xa517e0, 0xc0000262a0)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/XiaoMi/Gaea/proxy/plan.checkSQLs.func1(0xc0004a4390, 0xc000020750, 0xc00000e7c0)
/data/Gaea/proxy/plan/plan_test.go:151 +0x38a
github.com/XiaoMi/Gaea/proxy/plan.checkSQLs(0xc0004a4390, 0xc000020750, 0xa941c5)
/data/Gaea/proxy/plan/plan_test.go:160 +0x40
github.com/XiaoMi/Gaea/proxy/plan.getTestFunc.func1(0xc000490600)
/data/Gaea/proxy/plan/plan_test.go:109 +0x408
testing.tRunner(0xc000490600, 0xc000270fc0)
/usr/local/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:960 +0x350
FAIL github.com/XiaoMi/Gaea/proxy/plan 0.306s
ok github.com/XiaoMi/Gaea/proxy/router 0.017s coverage: 57.4% of statements
? github.com/XiaoMi/Gaea/proxy/sequence [no test files]
ok github.com/XiaoMi/Gaea/proxy/server 0.015s coverage: 3.2% of statements
ok github.com/XiaoMi/Gaea/stats 0.006s coverage: 69.6% of statements
ok github.com/XiaoMi/Gaea/stats/prometheus 0.036s coverage: 97.1% of statements
ok github.com/XiaoMi/Gaea/util 10.311s coverage: 49.3% of statements
ok github.com/XiaoMi/Gaea/util/bucketpool 0.054s coverage: 93.9% of statements
ok github.com/XiaoMi/Gaea/util/cache 0.001s coverage: 98.1% of statements
ok github.com/XiaoMi/Gaea/util/crypto 0.001s coverage: 69.7% of statements
ok github.com/XiaoMi/Gaea/util/hack 0.001s coverage: 55.0% of statements
? github.com/XiaoMi/Gaea/util/math [no test files]
ok github.com/XiaoMi/Gaea/util/requests 0.002s coverage: 48.5% of statements
ok github.com/XiaoMi/Gaea/util/sync2 0.027s coverage: 60.3% of statements
? github.com/XiaoMi/Gaea/util/testleak [no test files]
ok github.com/XiaoMi/Gaea/util/timer 0.702s coverage: 96.2% of statements
FAIL
make: *** [test] 错误 1

系统环境：

[root@ip-10-0-21-222 mod]# cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Golang版本
[root@ip-10-0-21-222 Gaea]# go version
go version go1.13 linux/amd64

同样的代码在Mac下可以正常通过，功能测试也正常

配置数据库从地址后，用户再配置成读写分离，所有的查询耗时都是150秒

slices.slaves配置后，users.rw_split再配置成1的话，所有的查询操作都很慢，150秒，rw_split改成0之后就正常了。

{
    "name": "biw_shard_min",
    "online": true,
    "read_only": false,
    "allowed_dbs": {
        "warehouse0901": true
    },
    "default_phy_dbs": {
    },
    "slices": [
        {
            "name": "slice-0",
            "user_name": "biw",
            "password": "密码不方便透露",
            "master": "rm-2ze68027fr85gw3ed-stresstest.mysql.rds.aliyuncs.com:3306",
            "slaves": ["rr-2ze4hz58a2rhmp45a-stresstests1.mysql.rds.aliyuncs.com:3306"],
            "capacity": 10,
            "max_capacity": 20,
            "idle_timeout": 60
        }
    ],
    "shard_rules": [
        
    ],
    "global_sequences": [],
    "users": [
        {
            "user_name": "biw_min",
            "password": "Hs7m8h2S0O4",
            "namespace": "biw_shard_min",
            "rw_flag": 2,
            "rw_split": 1,
            "other_property": 0
        }
    ],
    "default_slice": "slice-0"
}

编译报错

[root@cdbha_test4 Gaea]# make test
go test -coverprofile=.coverage.out ./...
ok github.com/XiaoMi/Gaea/backend 0.006s coverage: 3.4% of statements
? github.com/XiaoMi/Gaea/cc [no test files]
? github.com/XiaoMi/Gaea/cc/proxy [no test files]
? github.com/XiaoMi/Gaea/cc/service [no test files]
ok github.com/XiaoMi/Gaea/cmd/gaea 0.009s coverage: 14.0% of statements
? github.com/XiaoMi/Gaea/cmd/gaea-cc [no test files]
? github.com/XiaoMi/Gaea/core [no test files]
? github.com/XiaoMi/Gaea/core/errors [no test files]
? github.com/XiaoMi/Gaea/log [no test files]
? github.com/XiaoMi/Gaea/log/xlog [no test files]
ok github.com/XiaoMi/Gaea/models 0.008s coverage: 77.3% of statements
ok github.com/XiaoMi/Gaea/models/etcd 0.004s coverage: 4.4% of statements
? github.com/XiaoMi/Gaea/models/file [no test files]
ok github.com/XiaoMi/Gaea/mysql 0.440s coverage: 34.3% of statements
ok github.com/XiaoMi/Gaea/parser 0.129s coverage: 84.1% of statements
ok github.com/XiaoMi/Gaea/parser/ast 0.052s coverage: 57.1% of statements
ok github.com/XiaoMi/Gaea/parser/auth 0.004s coverage: 0.0% of statements [no tests to run]
ok github.com/XiaoMi/Gaea/parser/format 0.003s coverage: 81.3% of statements
ok github.com/XiaoMi/Gaea/parser/model 0.006s coverage: 53.6% of statements
ok github.com/XiaoMi/Gaea/parser/opcode 0.003s coverage: 55.6% of statements
? github.com/XiaoMi/Gaea/parser/stmtctx [no test files]
ok github.com/XiaoMi/Gaea/parser/terror 0.349s coverage: 75.9% of statements
ok github.com/XiaoMi/Gaea/parser/tidb-types 0.035s coverage: 67.3% of statements
ok github.com/XiaoMi/Gaea/parser/tidb-types/json 0.007s coverage: 81.4% of statements
ok github.com/XiaoMi/Gaea/parser/tidb-types/parser_driver 0.005s coverage: 24.7% of statements
? github.com/XiaoMi/Gaea/parser/types [no test files]
--- FAIL: TestSelectKingshardDateYear (0.00s)
--- FAIL: TestSelectKingshardDateYear/select_from_tbl_ks_year_where_create_time<=1388505600 (0.00s)
plan_test.go:110: not equal, expect: map[slice-0:map[db_ks:[SELECT * FROM tbl_ks_year_2014 WHERE create_time<=1388505600]]], actual: map[]
--- FAIL: TestSelectKingshardDateYear/selectfrom_tbl_ks_year_where_create_time<=1388505600#01 (0.00s)
plan_test.go:110: not equal, expect: map[slice-0:map[db_ks:[SELECT * FROM tbl_ks_year_2014 WHERE create_time<=1388505600]]], actual: map[]
--- FAIL: TestSelectKingshardDateYear/selectfrom_tbl_ks_year_where_create_time>=1514736000 (0.00s)
plan_test.go:110: not equal, expect: map[slice-1:map[db_ks:[SELECT * FROM tbl_ks_year_2018 WHERE create_time>=1514736000 SELECT * FROM tbl_ks_year_2019 WHERE create_time>=1514736000]]], actual: map[slice-0:map[db_ks:[SELECT * FROM tbl_ks_year_2017 WHERE create_time>=1514736000]] slice-1:map[db_ks:[SELECT * FROM tbl_ks_year_2018 WHERE create_time>=1514736000 SELECT * FROM tbl_ks_year_2019 WHERE create_time>=1514736000]]]
--- FAIL: TestSelectKingshardDateYear/selectfrom_tbl_ks_year_where_create_time>=_1577808000 (0.00s)
plan_test.go:110: not equal, expect: map[], actual: map[slice-1:map[db_ks:[SELECT * FROM tbl_ks_year_2019 WHERE create_time>=1577808000]]]
panic: runtime error: index out of range [1] with length 1 [recovered]
panic: runtime error: index out of range [1] with length 1

goroutine 410 [running]:
testing.tRunner.func1(0xc000516400)
/usr/lib/golang/src/testing/testing.go:874 +0x3a3
panic(0xa58d60, 0xc00012c600)
/usr/lib/golang/src/runtime/panic.go:679 +0x1b2
github.com/XiaoMi/Gaea/proxy/plan.checkSQLs.func1(0xc0000203c0, 0xc000020cc0, 0xc00000e9c0)
/root/Gaea/proxy/plan/plan_test.go:151 +0x38a
github.com/XiaoMi/Gaea/proxy/plan.checkSQLs(0xc0000203c0, 0xc000020cc0, 0xa9bb49)
/root/Gaea/proxy/plan/plan_test.go:160 +0x40
github.com/XiaoMi/Gaea/proxy/plan.getTestFunc.func1(0xc000516400)
/root/Gaea/proxy/plan/plan_test.go:109 +0x408
testing.tRunner(0xc000516400, 0xc000369f40)
/usr/lib/golang/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
/usr/lib/golang/src/testing/testing.go:960 +0x350
FAIL github.com/XiaoMi/Gaea/proxy/plan 0.213s
ok github.com/XiaoMi/Gaea/proxy/router 0.018s coverage: 57.4% of statements
? github.com/XiaoMi/Gaea/proxy/sequence [no test files]
ok github.com/XiaoMi/Gaea/proxy/server 0.014s coverage: 3.2% of statements
ok github.com/XiaoMi/Gaea/stats 0.005s coverage: 70.2% of statements
ok github.com/XiaoMi/Gaea/stats/prometheus 0.039s coverage: 97.1% of statements
ok github.com/XiaoMi/Gaea/util 10.316s coverage: 49.3% of statements
ok github.com/XiaoMi/Gaea/util/bucketpool 0.539s coverage: 93.9% of statements
ok github.com/XiaoMi/Gaea/util/cache 0.003s coverage: 98.1% of statements
ok github.com/XiaoMi/Gaea/util/crypto 0.004s coverage: 69.7% of statements
ok github.com/XiaoMi/Gaea/util/hack 0.002s coverage: 55.0% of statements
? github.com/XiaoMi/Gaea/util/math [no test files]
ok github.com/XiaoMi/Gaea/util/requests 0.020s coverage: 48.5% of statements
ok github.com/XiaoMi/Gaea/util/sync2 0.029s coverage: 60.3% of statements
? github.com/XiaoMi/Gaea/util/testleak [no test files]
ok github.com/XiaoMi/Gaea/util/timer 0.827s coverage: 96.2% of statements
FAIL
make: *** [test] Error 1

编译报错

go build -o /data/Gaea/bin/gaea -pkgdir=/pkg/linux_amd64 -ldflags "-extldflags -static -X /core.buildVersion=5fad791d48d4529fc4055437e8a6c7f057d07d9d-dirty -X /core.buildGitRevision=5fad791d48d4529fc4055437e8a6c7f057d07d9d-dirty -X /core.buildUser=root -X /core.buildHost=xxgk-qt-bf -X /core.buildStatus=Modified -X /core.buildTime=2020-03-04--04:44:07" /cmd/gaea go: go.etcd.io/[email protected]: unrecognized import path "go.etcd.io/bbolt" (https fetch: Get https://go.etcd.io/bbolt?go-get=1: dial tcp 172.217.24.19:443: connect: connection timed out) make: *** [gaea] Error 1

modify namespace failed case of read data in error

make报错

build github.com/XiaoMi/Gaea/cmd/gaea: cannot load github.com/gin-contrib/sse: Get https://athens.azurefd.net/github.com/gin-contrib/sse/@v/v0.0.0-20190301062529-5545eab6dad3.zip: stream error: stream ID 23; CANCEL

微信群满了，建议用钉钉

微信群满了，建议用钉钉

add cli tool to add、query、modify config of namespace

可以用这个实现行级权限吗？

完善namespace校验条件

提供插件机制

提供插件机制，比如sql审计和租户管理，用户可以按照规范，自行开发接口，编译成插件（比如so动态库）。

who uses gaea (gaea用户列表)

如果您的公司正在使用gaea, 欢迎更新此列表, 留下您的公司和网址. 谢谢您为gaea做出的贡献.

wrong database name in unshard mode

The database name is wrong when default_phy_dbs is set and run unshard SQL with db.table.

e.g.

namespace:

{
    "name": "gaea_namespace_1",
    "online": true,
    "read_only": true,
    "allowed_dbs": {
        "db_ks": true,
        "db_mycat": true
    },
    "default_phy_dbs": {
        "db_ks": "db_ks",
        "db_mycat": "db_mycat_0"
    },
    ....
}

SQL:

USE db_mycat;
SELECT * FROM db_mycat.unshard_table limit 10;

The backend database name is db_mycat_0, but unshard_table is not in sharding mode, so gaea-proxy cannot replace db_mycat with db_mycat_0, which lead to backend error SQL.

使用sysbench压测后stmt报错

sysbench命令为

sysbench /usr/local/Cellar/sysbench/1.0.16/share/sysbench/oltp_read_only.lua \
    --mysql-host=10.2.0.107 \
    --mysql-port=3306 \
    --mysql-user=root \
    --mysql-password=123456 \
    --threads=10 \
    --time=120 \
    --report-interval=10 \
    run

报错为

[2019-06-04 15:23:37] [gaea_proxy] [localhost] [WARN] [900000001] [github.com/XiaoMi/Gaea/proxy/server.(*Session).Run.func1:session.go:215] [server] Session Run panic error, error: runtime error: index out of range, stack: goroutine 40 [running]:
github.com/XiaoMi/Gaea/proxy/server.(*Session).Run.func1(0xc000380730)
	/Users/warm/go/src/github.com/XiaoMi/Gaea/proxy/server/session.go:213 +0x176
panic(0x1926bc0, 0x241bb30)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/XiaoMi/Gaea/proxy/server.(*SessionExecutor).bindStmtArgs(0xc0006ac000, 0xc0000c0a80, 0xc00067c20a, 0x1, 0x76, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/warm/go/src/github.com/XiaoMi/Gaea/proxy/server/executor_stmt.go:225 +0xeb9
github.com/XiaoMi/Gaea/proxy/server.(*SessionExecutor).handleStmtExecute(0xc0006ac000, 0xc00067c201, 0x13, 0x7f, 0x0, 0x0, 0x0)
	/Users/warm/go/src/github.com/XiaoMi/Gaea/proxy/server/executor_stmt.go:178 +0x1d4
github.com/XiaoMi/Gaea/proxy/server.(*SessionExecutor).ExecuteCommand(0xc0006ac000, 0xc0006cde17, 0xc00067c201, 0x13, 0x7f, 0x0, 0x0, 0x19ea000, 0xc000183420)
	/Users/warm/go/src/github.com/XiaoMi/Gaea/proxy/server/executor.go:295 +0x7a1
github.com/XiaoMi/Gaea/proxy/server.(*Session).Run(0xc000380730)
	/Users/warm/go/src/github.com/XiaoMi/Gaea/proxy/server/session.go:236 +0x2b2
github.com/XiaoMi/Gaea/proxy/server.(*Server).onConn(0xc000384f00, 0x1bc44e0, 0xc000010280)
	/Users/warm/go/src/github.com/XiaoMi/Gaea/proxy/server/server.go:139 +0x289
created by github.com/XiaoMi/Gaea/proxy/server.(*Server).Run
	/Users/warm/go/src/github.com/XiaoMi/Gaea/proxy/server/server.go:156 +0x17b

直接压测原生的mysql服务没有这个问题。

mysql 是 docker 版 5.7 version。

docker run -d --name sql \
    -e MYSQL_USER=root \
    -e MYSQL_ROOT_PASSWORD=123456 \
    -p 3307:3306 \
    mysql:5.7

replace glide with go modules

How about replace glide with go modules?

As an officical tool, go modules is easy enough to manage dependencies.

./bin/gaea 报错

init manager failed, error: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:2379: connect: connection refused

init manager failed, error: invalid character 'b' looking for beginning of value

如题，启动时报错，是bug吗，该怎么解决呢？这个报错没有定位，不好排查

multi cluster support in gaea-proxy

err: create select plan error: stmt type does not support shard now

这里有个小问题。我这里有用gaea替换掉在用的mycat的打算。体验过程中遇到点小问题。我这里下载 的是release v1.0.3.5月21号的版本。参考mycat schema.xml配置了gaea的分表规则。目前遇到两个小问题。1、namespace用户 连接 gaea后如果不加-A 会报ERROR 2006(HY000): MySQL server has gone away错误。（这个目前不影响select数据，可以使用） 2、我自己写了一个etc/file/namespace/*.json文件 加了一 个分表规则，我通过gaea建表的时候报err: create select plan error: stmt type does not support shard now。。。。json的分表规则我是这样写的。第二个问题导致我现在分表的情况下建不了表，不知道问题出在哪里。我咨询了一下社区的大神说是gaea目前不支持分表建表吗？

{
"name": "namespace_cas",
"online": true,
"read_only": false,
"allowed_dbs": {
"cas": true
},
"default_phy_dbs": {
"cas": "db_cas1"
},
"slices": [
{
"name": "slice-0",
"user_name": "cxs",
"password": "xxx",
"master": "172.16.1.22:3306",
"capacity": 128,
"max_capacity": 128,
"idle_timeout": 3600
},
{
"name": "slice-1",
"user_name": "cxs",
"password": "xxx",
"master": "172.16.1.46:3306",
"capacity": 128,
"max_capacity": 128,
"idle_timeout": 3600
}
],
"shard_rules": [
{
"db": "cas",
"table": "TEST_T_LAS_ROLLOVER_BOOK",
"type": "mycat_murmur",
"key": "CUST_NO",
"locations": [
1,
1
],
"slices": [
"slice-0",
"slice-1"
],
"databases": [
"db_cas1",
"db_cas2"
],
"seed": "0",
"virtual_bucket_times": "160"
}
],
"users": [
{
"user_name": "cas",
"password": "cas",
"namespace": "namespace_cas",
"rw_flag": 2,
"rw_split": 1
}
],
"default_slice": "slice-0"
}

making test target executed failed

make test executed failed in local environment.

some logs as follow:

----------------------------------------------------------------------
FAIL: terror_test.go:109: testTErrorSuite.TestTraceAndLocation

terror_test.go:114:
    c.Assert(len(lines), Equals, 23)
... obtained int = 21
... expected int = 23

OOPS: 4 passed, 1 FAILED
--- FAIL: TestT (0.00s)
FAIL
coverage: 75.9% of statements
FAIL    github.com/XiaoMi/Gaea/parser/terror    0.027s

Gaea支持类似于Kingshard中的强制读主库功能吗?

在对数据一致性要求比较高的场景下,kingshard的强制读主库功能还是很有用的,不知道Gaea是否有此功能

glide up 依赖更新后，编译报错

make 编译报错如下：

执行过程如下：
glide update & glide install
make

range路由结果错误

multi cluster support in gaea-cc

监控模块生成每个时段的火焰图能分享下吗?

之前在gopher上看到了监控那块,能查看每个时段的火焰图?是定时把profile存下来吗?能分享下吗?

该项目在持续维护吗

Gaea有没有和别的中间件在性能方面的对比数据?

如何清理慢sql指纹日志？

将慢sql和错误sql解决后，如何删除盖亚上的相关日志？目前每次通过接口都能获取自盖亚启动以来所有的相关指纹。

需求：监控问题sql指纹，但解决后需要清除数据，以便查看解决后是否有问题。

wait_timeout 设置

mysql设置了wait_timeout 比较小，gaea需要怎么设置和mysql的wait_timeout相匹配