jd-opensource/vtdriver

VtDriver

English Docs

VtDriver是一套基于分布式数据库Vitess而开发的Vitess Java客户端解决方案，在Java的JDBC层实现了Vitess关于分库、健康检查、灵活水平拆分和主从切换等额外服务。VtDriver通过jar包的形式提供服务，帮助应用程序直连数据库而非使用中间代理（VTGate）节点。因为完全基于标准的JDBC接口来实现的，VtDriver可以理解为Vitess的JDBC驱动，兼容JDBC、大部分主流数据库链接池和ORM框架。

Content Table

• Why VtDriver
• Architecture
• Quick Start
• Features/Configuration
• Road Map
• Acknowledgements

Why VtDriver

VtDriver是对Vitess生态的一种解决方案补充，旨在缩短数据访问链路、提高性能和节约资源，从而达到节能增效的目的，适用于Java开发的对性能有极高要求的OLTP应用和对资源消耗有极其严苛限制的场景，以下图为例：

如图左所示，作为一款分布式数据库，Vitess开发了一些中间代理模块来解决异构语言、集群管理/保护、数据同步和适配多数据引擎等问题。这样的设计能够为数据库带来高可用、灵活性和扩展性。但出于提升数据库性能，降低访问时延和节约资源的考虑，VtDriver在最消耗资源的sql请求和数据返回的链路上做出了改变，如上图所示。

相比于Vitess，一个数据请求（DML）从应用端发出到接收，需要经过 app->VTGate->VTTablet->database->VTTablet->VTGate-> app的路径，VtDriver通过jar包的形式将VTGate的能力（分布式SQL改写、结果收集处理、健康检查/切换）赋能给了各应用程序节点，使得应用程序能够直接访问数据库app ->database （图右所示），从而减少了网络上的开销。并且，因为降低了对于中间节点的依赖，可以不再像原来一样需要投入大量资源来维护中间节点的稳定性，对于那些非常在意CPU资源消耗的使用者来说，更加合适。

需要注意的是，VtDriver是Vitess生态的补充方案而非替代方案，VtDriver仍然需要与VTTablet，Topo元数据等进行交互，来完成健康检查、故障自动切换和横向扩缩容等功能。因此在部署上仍然需要先搭建一套完整的Vitess架构。此外，因为跳过了VTTablet直连数据库，在某些情况下也会造成数据库链接消耗过大的情况，这也需要使用者在使用时注意。以下是使用Vitess传统链接方式VTGate和VtDriver方式所带来的一些特点的比较：

	Vitess-VTGate	Vitess-VtDriver
链接数消耗	低	高
异构语言	任意	仅Java
性能	损耗略高	损耗低
客户端入口	有	无

Architecture

如图所示，是VtDriver的设计架构，主要分为接口API模块，查询计划生成的Plan模块，负责执行和计算的Engine模块，负责事务的TX模块，数据库底层连接的Datasource模块和分布式数据库管理的Topo模块：

• API：实现了JDBC标准接口模块，Java用户只需要替换驱动，代码不需要做任何修改便可使用VtDriver
• Plan：基于SQL请求和数据库Topo元数据生成相应的执行计划
• Engine：负责执行Plan生成的执行计划，并收集和计算各个分片返回的结果
• TX：负责单库时候和多库执行时候的事务管理， 严格的两阶段提交功能目前还在开发当中
• Datasource：负责与各个分片集群的真实数据库的连接，包括了连接池管理，驱动管理（目前仅支持MySQL）和配置管理
• Topo Management ：负责读取和管理分布式数据库的Topo信息和元数据，包括了Tablet信息和状态、数据库表信息和分表Index，实时健康检查，并负责在监测到Tablet异常、主从切换或resharding完成的时候执行响应操作

Quick Start

• 可参考VtDriver环境搭建

Features/Configuration

因为是基于Vitess开发出来的Java 驱动客户端，因此VtDriver在很多方面保持了跟Vitess一样的特性，比如分片执行计划/引擎、事物管理、topo和健康管理等。不过因为VtDriver是以Java JDBC驱动为目的开发的，导致其在某些方面的设计与以数据库服务器为特性开发的Vitess VTGate有些许不同，并且大部分配置信息通过url进行配置。另外，基于业务等需求等原因，我们也在Vitess上面加了些许新的功能。

Split Table：

分表是VtDriver开发的一个新功能点，基于现有的分片执行计划/引擎基础开发而来，主要针对单表数据量过大而影响性能的问题。具体可查看VtDriver分表

Read-Write Splitting：

不同于Vitess读写分离，VtDriver通过在datasource的Url中配置role=rr/rw来确定读写分离，默认为读主。

Consolidation：

Query Consolidation是Vitess在VTTablet层开发的一个用以保护数据库不受某个特定慢查询反复同时执行拖累而过载的一个功能，VtDriver将其移植到了驱动层。其原理是当一个SQL请求发给VtDriver，VtDriver监测到其一样的一个SQL请求还在执行当中时，会将当前SQL请求暂存，等到其前一个SQL请求的返回结果，并将其返回结果同时也当作当前SQL的返回结果返回。

与读写分离一样，该功能同样需要其在datasource的Url中配置queryConsolidator=true。

Specified Shard Destination

VtDriver/Vitess通过解析SQL语句中的分片键与值来判断SQL需要被发往哪个分片的。但是有些时候，分片条件并不存在于SQL，而存在于外部业务逻辑。因此，VtDriver允许使用者通过发送sql带注释的方式，在执行计划前解析注释中set_shard指令来确定是否需要指定分片执行，例如：/*shard=-80*/ select * from table。

需要注意的是，出于水平扩展的考虑，指定分片只支持update\delete\select，不支持insert。另外，如果数据库进行了水平拆分，需要修改指定shard范围，否则会报错。

VtDriver支持配置的参数

• VtDriver支持配置的参数

Road Map

Database

• Vitess based on MySQL （其他数据库产品，直连MySQL暂不支持）

VIndex

• 仅支持murmurHash Vitess的分片算法目前暂未实现

DML/DDL

• Select/Insert/Update/Delete
• 部分SET语句
• SHOW语句暂不支持
• DDL语句暂不支持

SQL

• AGGREGATION
• DISTINCT
• HAVING
• ORDER BY
• GROUP BY
• JOIN
• UNION / UNION ALL
• SUBQUERY / PULLOUT SUBQUERY

详细内容请参考 VtDriver支持的SQL。

Query Method

• Simple Query
• Stream Query
• MultiQuery (JOIN/UNION/PULLOUT SUBQUERY不支持)

Auto Increment Key

• Sequence （暂不支持）

Distributed Transactions

• best-effort distributed transactions
• 暂不支持2PC (two-phase commit)

Acknowledgements

VtDriver在开发中借鉴了许多开源的程序、框架的方法和意见，在此做出感谢，是他们让我们站在了巨人的肩膀上去眺望世界：

• Vitess
• Druid
• mysql-connector
• prometheus
• HikariCp
• ShardingSphere

Note: 以上特别感谢Vitess，Vitess给我们节省大量研发成本，感谢PlanetScale CTO Sugu Sougoumarane 对我们团队的支持。

vtdriver's Issues

Support reading from the master database when the replica is unavailable in a read-write separation scenario

Description

In a read-write separation scenario, where a database system is configured with separate master and replica databases for read and write operations respectively, there is a need to handle situations when the replica database becomes unavailable. Currently, the system does not support automatically routing read requests to the master database when the replica is inaccessible.

It takes a long time to establish a connection when the application start

When the read-write separation function is used (role=rr is configured in the url), it takes a long time (more than 10s) to establish a connection for the first time.

There may be the following problems with loadTablets in TopologyWatcher:

1. VtHealthCheckExecutorService, its cole-pool-size is set using Runtime.getRuntime().availableProcessors() method. This method obtains the number of cores of the physical machine in the early jdk version, and after Java SE 8u131 and JDK 9, the method obtains the number of cores of the container.

2. If there are a lot of Tablets in the cell (for example, there are thousands of Tablets), but the user only cares about one or two of them, the current method is to obtain all the Tablets in the cell one by one in a concurrent way for matching. This process consumes a lot of time.

Monitor metrics optimization

1. add a metrics plan_cache_size
2. change uid to "shard+ip", currently is tablet_alias

In the scenario of read-write separation, initialization takes a long time

NullPointerException when executing the following SQL query: select id, name, age from t_users where id = ? and name < ? and age in (select 1)

Problem Description:

When executing the SQL query mentioned below, a NullPointerException occurs:

select id, name, age
from t_users
where id = ? and name < ? and age in (select 1)

Error Message:

java.lang.NullPointerException
	at com.jd.jdbc.sqlparser.SqlParser.combineRepeatBindVars(SqlParser.java:614)
	at com.jd.jdbc.sqlparser.SqlParser.rewriteAst(SqlParser.java:587)
	at com.jd.jdbc.sqlparser.SqlParser.prepareAst(SqlParser.java:560)
	at com.jd.jdbc.Executor.getPlan(Executor.java:292)
	at com.jd.jdbc.Executor.newExecute(Executor.java:438)
	at com.jd.jdbc.Executor.execute(Executor.java:138)
	at com.jd.jdbc.vitess.VitessStatement.executeInternal(VitessStatement.java:447)
	at com.jd.jdbc.vitess.VitessPreparedStatement.execute(VitessPreparedStatement.java:418)

Steps to Reproduce:

Execute the provided SQL query mentioned above.
The error message indicates a java.lang.NullPointerException exception.

Expected Result:

The SQL query should be executed successfully without encountering a NullPointerException.

When using &allowMultiQueries=true, there may be problems when sql is split

protected List<String> split(String sql) throws SQLException {
        List<String> sqls = new ArrayList<>(Splitter.on(";").omitEmptyStrings().trimResults().splitToList(sql));
        if (sqls.isEmpty()) {
            throw new SQLException("trying to execute empty queries " + sqls);
        }

        return sqls;
    }

When sql needs to be split, the method used in VtDriver is similar to sql.split(";"). When ";" exists in comment or value, sql will be split into unexpected results.
Considering the performance loss of using syntax analysis, a tool for splitting SQL can be implemented.

SQLFeatureNotSupportedException: not supported java.math.BigInteger

org.springframework.dao.InvalidDataAccessApiUsageException: PreparedStatementCallback; SQL [SELECT * FROM `t` WHERE f_key_id = ?]; not supported java.math.BigInteger; nested exception is java.sql.SQLFeatureNotSupportedException: not supported java.math.BigInteger

	at org.springframework.jdbc.support.SQLExceptionSubclassTranslator.doTranslate(SQLExceptionSubclassTranslator.java:96)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:70)
	at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:79)
	at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1541)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:757)
	at org.springframework.jdbc.core.JdbcTemplate.queryForObject(JdbcTemplate.java:886)
	at com.jd.jdbc.vtdriver.demo.JdbcTemplateTest.select2(JdbcTemplateTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:221)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
Caused by: java.sql.SQLFeatureNotSupportedException: not supported java.math.BigInteger
	at com.jd.jdbc.vitess.VitessPreparedStatement.setObject(VitessPreparedStatement.java:396)
	at com.jd.zaxxer.hikari.pool.HikariProxyPreparedStatement.setObject(HikariProxyPreparedStatement.java)
	at org.springframework.jdbc.core.StatementCreatorUtils.setValue(StatementCreatorUtils.java:415)
	at org.springframework.jdbc.core.StatementCreatorUtils.setParameterValueInternal(StatementCreatorUtils.java:237)
	at org.springframework.jdbc.core.StatementCreatorUtils.setParameterValue(StatementCreatorUtils.java:168)
	at org.springframework.jdbc.core.ArgumentPreparedStatementSetter.doSetValue(ArgumentPreparedStatementSetter.java:69)
	at org.springframework.jdbc.core.ArgumentPreparedStatementSetter.setValues(ArgumentPreparedStatementSetter.java:50)
	at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:720)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651)
	... 28 more

syntax parsing error in the 'select sql'

Log4J2漏洞问题，短期内持续关注一下吧

2.16.0版本有新的漏洞爆出，CVE-2021-45105。

Refactor RoutePlan, TableRoutePlan

Lazy call to VitessConnection.buildMetaDataInfo() method

inconsistent metadata in VtDriver memory when VTTablet goes offline.

Incorrect metadata returned by VitessResultSet

Details

1. join subquery

select id, t.id from ( select user.id from user join user_extra on user.id < 3) as t

Second field t.id missed.

2. Stream query

select * from user

In streaming query case, by invoking ResultSet.next() for a few times, buildResult() would fail when cached result data has been ran out.

3. lastInsertId

select last_insert_id()
select @@IDENTITY

Empty metadata.

4. select ... from dual

"select 10"
"select 2 as t"
// "select 'abc字符串'"
"select 1 + 5"
"select 1 + 5.0" // INT+FLOAT
"select 1.0 + 5.0" // FLOAT + FLOAT
"select 6.0"
"select 1.502053"
"select 10.5025"
"select 0.0050"
"select 2/3 as result from dual" // INT/INT = FLOAT

Metadata are different with mysql connectorj's.

5. select Where Id=Null

select id, name, textcol1 from user where textcol1=NULL

6. multiquery

select id, name, textcol1 from user where textcol1 = NULL; select 1 from user

7. Stream+union

select id from user_extra union all select id from user_metadata
column 1 getTableName: expected user_extra, but user_metadata

Will return left or right subquery's tableName, depending on which subquery is returned first.

Having Filter parameters are not normalized

SELECT * FROM T HAVING a=1. after normalize is still SELECT * FROM T HAVING a=1

In Druid, HAVING and GROUP BY are both subnodes under SQLSelectGroupByClause. GROUP BY will have 1, 2, and 3, so normalizer will not be extracted. However, this is associated with having and does not carry out parameter extraction

Caused by: java.lang.NumberFormatException: For input string: "!u"

Caused by: java.sql.SQLException: 
	at com.jd.jdbc.exception.SQLExceptionTranslator.translate(SQLExceptionTranslator.java:76)
	at com.jd.jdbc.queryservice.NativeQueryService.execute(NativeQueryService.java:150)
	at com.jd.jdbc.queryservice.CombinedQueryService.execute(CombinedQueryService.java:121)
	at com.jd.jdbc.queryservice.RetryTabletQueryService.lambda$execute$3(RetryTabletQueryService.java:141)
	at com.jd.jdbc.queryservice.RetryTabletQueryService.retry(RetryTabletQueryService.java:306)
	at com.jd.jdbc.queryservice.RetryTabletQueryService.execute(RetryTabletQueryService.java:139)
	at com.jd.jdbc.srvtopo.ScatterConn.lambda$executeMultiShard$0(ScatterConn.java:143)
	at com.jd.jdbc.srvtopo.ScatterConn.oneShard(ScatterConn.java:456)
	at com.jd.jdbc.srvtopo.ScatterConn.multiGoTransaction(ScatterConn.java:325)
	at com.jd.jdbc.srvtopo.ScatterConn.executeMultiShard(ScatterConn.java:118)
	at com.jd.jdbc.Executor.executeMultiShard(Executor.java:208)
	at com.jd.jdbc.VcursorImpl.executeMultiShard(VcursorImpl.java:103)
	at com.jd.jdbc.engine.RouteEngine.exec(RouteEngine.java:238)
	at com.jd.jdbc.engine.RouteEngine.execute(RouteEngine.java:130)
	at com.jd.jdbc.Executor.lambda$executePlan$0(Executor.java:346)
	at com.jd.jdbc.Executor.newExecute(Executor.java:485)
	at com.jd.jdbc.Executor.execute(Executor.java:138)
	at com.jd.jdbc.vitess.VitessStatement.executeInternal(VitessStatement.java:434)
	at com.jd.jdbc.vitess.VitessPreparedStatement.execute$original$otrY9uYG(VitessPreparedStatement.java:413)
	at com.jd.jdbc.vitess.VitessPreparedStatement.execute$original$otrY9uYG$accessor$jrWYjC60(VitessPreparedStatement.java)
	at com.jd.jdbc.vitess.VitessPreparedStatement$auxiliary$qMFRLO3q.call(Unknown Source)
	at com.jd.pfinder.profiler.tracer.plugin.invoker.InstanceEnhanceInvoker.intercept(InstanceEnhanceInvoker.java:62)
	at com.jd.jdbc.vitess.VitessPreparedStatement.execute(VitessPreparedStatement.java)
	at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3409)
	at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_execute(FilterEventAdapter.java:440)
	at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_execute(FilterChainImpl.java:3407)
	at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.execute(PreparedStatementProxyImpl.java:167)
	at com.alibaba.druid.pool.DruidPooledPreparedStatement.execute(DruidPooledPreparedStatement.java:497)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:63)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
	at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:326)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
	at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at com.sun.proxy.$Proxy314.query(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
	at com.sun.proxy.$Proxy314.query(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
	at com.sun.proxy.$Proxy314.query(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at com.sun.proxy.$Proxy314.query(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
	at com.sun.proxy.$Proxy314.query(Unknown Source)
	at com.github.pagehelper.PageInterceptor.intercept(PageInterceptor.java:143)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at com.sun.proxy.$Proxy314.query(Unknown Source)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:77)
	at sun.reflect.GeneratedMethodAccessor425.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
	... 99 common frames omitted
Caused by: java.sql.SQLException: 162
	at com.jd.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
	at com.jd.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	at com.jd.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.jd.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:764)
	at com.jd.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:781)
	at com.jd.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:102)
	at com.jd.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
	at com.jd.jdbc.pool.InnerConnection.execute(InnerConnection.java:47)
	at com.jd.jdbc.queryservice.NativeQueryService.execute(NativeQueryService.java:145)
	... 170 common frames omitted

Caused by: java.sql.SQLException: Error
	at com.alibaba.druid.pool.DruidDataSource.handleConnectionException(DruidDataSource.java:1652)
	at com.alibaba.druid.pool.DruidPooledConnection.handleException(DruidPooledConnection.java:133)
	at com.alibaba.druid.pool.DruidPooledStatement.checkException(DruidPooledStatement.java:82)
	at com.alibaba.druid.pool.DruidPooledResultSet.checkException(DruidPooledResultSet.java:55)
	at com.alibaba.druid.pool.DruidPooledResultSet.getTimestamp(DruidPooledResultSet.java:368)
	at org.apache.ibatis.type.DateTypeHandler.getNullableResult(DateTypeHandler.java:39)
	at org.apache.ibatis.type.DateTypeHandler.getNullableResult(DateTypeHandler.java:28)
	at org.apache.ibatis.type.BaseTypeHandler.getResult(BaseTypeHandler.java:66)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getPropertyMappingValue(DefaultResultSetHandler.java:471)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.applyPropertyMappings(DefaultResultSetHandler.java:440)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:403)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:355)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:330)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:303)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:196)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
	at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:326)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
	at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
	at com.sun.proxy.$Proxy317.query(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at com.sun.proxy.$Proxy317.query(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
	at com.sun.proxy.$Proxy317.query(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:63)
	at com.sun.proxy.$Proxy317.query(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.ibatis.plugin.Invocation.proceed(Invocation.java:49)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at com.sun.proxy.$Proxy317.query(Unknown Source)
	at com.github.pagehelper.PageInterceptor.intercept(PageInterceptor.java:143)
	at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
	at com.sun.proxy.$Proxy317.query(Unknown Source)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:141)
	at sun.reflect.GeneratedMethodAccessor329.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433)
	... 165 common frames omitted
Caused by: java.lang.NumberFormatException: For input string: "!u"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:569)
	at java.lang.Integer.parseInt(Integer.java:615)
	at com.jd.jdbc.vitess.resultset.DateTimeUtil$InternalTimestamp.parseInternalTimestamp(DateTimeUtil.java:276)
	at com.jd.jdbc.vitess.resultset.DateTimeUtil.getTimestampFromTimestampString(DateTimeUtil.java:103)
	at com.jd.jdbc.vitess.resultset.ResultSetUtil.convertToTimestamp(ResultSetUtil.java:709)
	at com.jd.jdbc.vitess.resultset.ResultSetUtil.convertValue(ResultSetUtil.java:108)
	at com.jd.jdbc.vitess.VitessResultSet.getTimestamp(VitessResultSet.java:333)
	at com.jd.jdbc.vitess.VitessResultSet.getTimestamp(VitessResultSet.java:339)
	at com.alibaba.druid.filter.FilterChainImpl.resultSet_getTimestamp(FilterChainImpl.java:1258)
	at com.alibaba.druid.filter.FilterAdapter.resultSet_getTimestamp(FilterAdapter.java:1819)
	at com.alibaba.druid.filter.FilterChainImpl.resultSet_getTimestamp(FilterChainImpl.java:1254)
	at com.alibaba.druid.proxy.jdbc.ResultSetProxyImpl.getTimestamp(ResultSetProxyImpl.java:741)
	at com.alibaba.druid.pool.DruidPooledResultSet.getTimestamp(DruidPooledResultSet.java:366)
	... 219 common frames omitted

Failed to connect VTTablet

Vitess 0.12, and keep getting errors:

Nov 26, 2021 3:51:48 PM io.vitess.shaded.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: ~* Channel ManagedChannelImpl{logId=23354, target=10.224.72.227:15999} was not shutdown properly!!! *~
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at io.vitess.shaded.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.(ManagedChannelOrphanWrapper.java:93)
at io.vitess.shaded.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:53)
at io.vitess.shaded.io.grpc.internal.ManagedChannelOrphanWrapper.(ManagedChannelOrphanWrapper.java:44)
at io.vitess.shaded.io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:596)
at io.vitess.shaded.io.grpc.ForwardingChannelBuilder.build(ForwardingChannelBuilder.java:255)
at com.jd.jdbc.queryservice.TabletDialer.dial(TabletDialer.java:49)
at com.jd.jdbc.discovery.TabletHealthCheck.getTabletQueryServiceLocked(TabletHealthCheck.java:132)
at com.jd.jdbc.discovery.TabletHealthCheck.getHealthCheckQueryService(TabletHealthCheck.java:121)
at com.jd.jdbc.discovery.TabletHealthCheck.startHealthCheckStream(TabletHealthCheck.java:217)
at com.jd.jdbc.discovery.HealthCheck$1.lambda$run$0(HealthCheck.java:410)
at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
at com.jd.jdbc.discovery.HealthCheck$1.run(HealthCheck.java:383)
at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
at java.base/java.util.TimerThread.run(Timer.java:506)

not supported DatabaseMetaDataResultSet.getStatement

Caused by: java.sql.SQLFeatureNotSupportedException
	at com.jd.jdbc.vitess.resultset.AbstractDatabaseMetaDataResultSet.getStatement(AbstractDatabaseMetaDataResultSet.java:96)

update jetcd version

Optimize startup time

Currently, the startup takes a long time

Null Pointer Exception when updating user with subquery in UPDATE statement

I am encountering a null pointer exception when running the following SQL query to update the name of a user based on a subquery:

UPDATE t_user a, (SELECT ? AS id, ? AS name) b SET a.name = b.name WHERE a.id = b.id

when I run this query, I get a null pointer exception. Upon further investigation, I found that the subquery is returning null and causing the issue.

Consolidator keeps reading expired response In high concurrency scenarios

In the existing implementation of consolidator, "queryCount" of the consolidator result will not be cleared for a long time in the case of high concurrency, the subsequent connection will continue to obtain outdated data.

Solution
Consider remove the mechanism of counting query

Verify the servertimezone configured by the application

1. Ensure that the application is configured with the correct serverTimeZone to avoid data inconsistency between applications.
2. Urlparser throws an inappropriate exception when no value is specified for the jdbcUrl parameter.

Refactor jetcd watch

Description:

The current implementation of jetcd watch logic is difficult to read and maintain. The code is complex and has too many nested loops, which makes it hard to understand the flow of execution. To improve the code, we propose refactoring the watch logic to use a thread pool and listener pattern.

The thread pool will allow us to handle multiple watch events concurrently, improving the system's efficiency and reducing latency. Meanwhile, the listener pattern will help decouple the watch logic from the rest of the system, making it more modular and easier to maintain.

Some possible steps to implement this refactoring include:

• Creating a thread pool to handle watch events
• Refactoring the watch logic to use a listener pattern
• Updating the code to use the new thread pool and listener pattern
• Adding tests to ensure that the refactored code works correctly

Null Pointer Exception happen occasionally when app start

Use GetTabletsByCell in healthcheck

vitessio/vitess#14693

Support omitting the "cell" parameter in the database connection jdbcurl

Description:

Currently, when connecting to a database using a connection URL, the "cell" parameter is required to specify the specific cell within a database cluster. However, in certain scenarios, it would be beneficial to allow the omission of the "cell" parameter.

To enhance the usability and flexibility of the database connection process, it is proposed to modify the system to support connecting to the default cell when the "cell" parameter is omitted from the connection jdbcurl. This means that users can establish a connection without explicitly specifying the cell, relying on the system's default behavior to determine the appropriate cell.

automatic rollback when multiquery getting execption in transaction

TabletAlias parsing error NumberFormatException

VitessResultSet部分类型转换不正确

• bit(1) bit(4) tinyint(1) 在getBytes出错;
• 时间类型 在getBytes,getBlob出错;
• 涉及到inputStreamReader编码格式会出错;
• bit(1) bit(4) getBlob出错；

Cross Shard Join report NPE exception

SQL：

SELECT a.*
FROM organization a
	JOIN distributioncenter b
	ON a.orgId = b.orgId
WHERE a.typeId = 1

Table：

//vindex: organizationId
CREATE TABLE `ufo_o_organization` (
	`organizationId` bigint(10) NOT NULL AUTO_INCREMENT COMMENT ',
	`orgId` int(10) unsigned NOT NULL COMMENT '',
	PRIMARY KEY (`organizationId`),
) ENGINE = InnoDB AUTO_INCREMENT = 814 DEFAULT CHARSET = utf8;


//vindex: disCenterId
CREATE TABLE `ufo_o_distributioncenter` (
	`disCenterId` bigint(10) NOT NULL AUTO_INCREMENT COMMENT '',
	`dcId` int(10) NOT NULL COMMENT '',
	`dcName` varchar(50) NOT NULL COMMENT '',
	`orgid` int(10) DEFAULT NULL COMMENT '',
	PRIMARY KEY (`disCenterId`),
	UNIQUE KEY `uniq_ufo_o_distributioncenter` (`dcId`, `orgid`, `yn`)
) ENGINE = InnoDB AUTO_INCREMENT = 1179 DEFAULT CHARSET = utf8 ;

Error:

java.lang.NullPointerException
 at com.jd.jdbc.sqlparser.dialect.mysql.visitor.VtRestoreVisitor.visit(VtRestoreVisitor.java:166)
 at com.jd.jdbc.sqlparser.visitor.SQLASTOutputVisitor.printExpr(SQLASTOutputVisitor.java:950)
 at com.jd.jdbc.sqlparser.visitor.SQLASTOutputVisitor.visitorBinaryRight(SQLASTOutputVisitor.java:863)
 at com.jd.jdbc.sqlparser.visitor.SQLASTOutputVisitor.visit(SQLASTOutputVisitor.java:817)
 at com.jd.jdbc.sqlparser.visitor.SQLASTOutputVisitor.printExpr(SQLASTOutputVisitor.java:938)
 at com.jd.jdbc.sqlparser.visitor.SQLASTOutputVisitor.visitBinaryLeft(SQLASTOutputVisitor.java:896)
 at com.jd.jdbc.sqlparser.visitor.SQLASTOutputVisitor.visit(SQLASTOutputVisitor.java:784)
 at com.jd.jdbc.sqlparser.visitor.SQLASTOutputVisitor.printExpr(SQLASTOutputVisitor.java:938)
 at com.jd.jdbc.sqlparser.dialect.mysql.visitor.MySqlOutputVisitor.visit(MySqlOutputVisitor.java:185)
 at com.jd.jdbc.sqlparser.dialect.mysql.ast.statement.MySqlSelectQueryBlock.accept0(MySqlSelectQueryBlock.java:222)
 at com.jd.jdbc.sqlparser.dialect.mysql.ast.statement.MySqlSelectQueryBlock.accept0(MySqlSelectQueryBlock.java:213)
 at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:41)
 at com.jd.jdbc.engine.Engine.getQueries(Engine.java:177)
 at com.jd.jdbc.engine.RouteEngine.exec(RouteEngine.java:270)
 at com.jd.jdbc.engine.RouteEngine.execute(RouteEngine.java:175)
 at com.jd.jdbc.engine.JoinEngine.execute(JoinEngine.java:120)
 at com.jd.jdbc.engine.JoinEngine.execute(JoinEngine.java:95)
 at com.jd.jdbc.engine.JoinEngine.execute(JoinEngine.java:95)
 at com.jd.jdbc.Executor.lambda$executePlan$0(Executor.java:338)
 at com.jd.jdbc.Executor.newExecute(Executor.java:489)
 at com.jd.jdbc.Executor.execute(Executor.java:130)
 at com.jd.jdbc.vitess.VitessStatement.executeInternal(VitessStatement.java:433)
 at com.jd.jdbc.vitess.VitessStatement.execute(VitessStatement.java:765)
 at com.jd.zaxxer.hikari.pool.ProxyStatement.execute(ProxyStatement.java:94)
 at com.jd.zaxxer.hikari.pool.HikariProxyStatement.execute(HikariProxyStatement.java)
 at java_sql_Statement$execute.call(Unknown Source)

NPE occurred when target Changed

When VTTablet Reparent

VtDriver will print following error message:

at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427) ~[mybatis-spring-2.0.6.jar:2.0.6]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source) ~[?:?]
at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:181) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194) ~[mybatis-3.5.7.jar:3.5.7]
at com.sun.proxy.$Proxy207.update(Unknown Source) ~[?:?]
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:64) ~[mybatis-3.5.7.jar:3.5.7]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source) ~[?:?]
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) ~[mybatis-3.5.7.jar:3.5.7]
at com.sun.proxy.$Proxy208.execute(Unknown Source) ~[?:?]
at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:59) ~[mybatis-3.5.7.jar:3.5.7]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at sun.reflect.GeneratedMethodAccessor192.invoke(Unknown Source) ~[?:?]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-3.4.2.jar:?]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-3.4.2.jar:?]
at com.jd.jdbc.vitess.VitessPreparedStatement.execute(VitessPreparedStatement.java:413) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.vitess.VitessStatement.executeInternal(VitessStatement.java:433) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.execute(Executor.java:132) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.newExecute(Executor.java:484) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.insideTransaction(Executor.java:689) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.lambda$executePlan$0(Executor.java:340) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.engine.InsertEngine.execute(InsertEngine.java:212) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.engine.InsertEngine.execInsertSharded(InsertEngine.java:310) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.VcursorImpl.executeMultiShard(VcursorImpl.java:100) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.executeMultiShard(Executor.java:202) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.srvtopo.ScatterConn.executeMultiShard(ScatterConn.java:118) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.srvtopo.ScatterConn.multiGoTransaction(ScatterConn.java:325) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.srvtopo.ScatterConn.oneShard(ScatterConn.java:456) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.srvtopo.ScatterConn.lambda$executeMultiShard$0(ScatterConn.java:156) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.queryservice.RetryTabletQueryService.beginExecute(RetryTabletQueryService.java:112) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.queryservice.RetryTabletQueryService.retry(RetryTabletQueryService.java:304) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.queryservice.RetryTabletQueryService.lambda$beginExecute$2(RetryTabletQueryService.java:114) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.queryservice.CombinedQueryService.beginExecute(CombinedQueryService.java:162) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.queryservice.CombinedQueryService.getNativeQueryService(CombinedQueryService.java:82) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.queryservice.NativeQueryService.<init>(NativeQueryService.java:84) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.pool.StatefulConnectionPool.getStatefulConnectionPool(StatefulConnectionPool.java:72) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.pool.StatefulConnectionPool.<init>(StatefulConnectionPool.java:56) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.pool.HikariUtil.getHikariConfig(HikariUtil.java:51) ~[vtdriver-1.0.0.jar:?]
at com.jd.zaxxer.hikari.HikariConfig.setDataSourceProperties(HikariConfig.java:468) ~[HikariCP-1.0-20210311.110649-1.jar:?]
at java.util.Hashtable.putAll(Hashtable.java:522) ~[?:1.8.0_60]
Caused by: java.lang.NullPointerException
... 155 more
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:427) ~[mybatis-spring-2.0.6.jar:2.0.6]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source) ~[?:?]
at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:181) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:194) ~[mybatis-3.5.7.jar:3.5.7]
at com.sun.proxy.$Proxy207.update(Unknown Source) ~[?:?]
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:64) ~[mybatis-3.5.7.jar:3.5.7]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source) ~[?:?]
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74) ~[mybatis-3.5.7.jar:3.5.7]
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47) ~[mybatis-3.5.7.jar:3.5.7]
at com.sun.proxy.$Proxy208.execute(Unknown Source) ~[?:?]
at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:59) ~[mybatis-3.5.7.jar:3.5.7]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at sun.reflect.GeneratedMethodAccessor192.invoke(Unknown Source) ~[?:?]
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-3.4.2.jar:?]
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-3.4.2.jar:?]
at com.jd.jdbc.vitess.VitessPreparedStatement.execute(VitessPreparedStatement.java:413) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.vitess.VitessStatement.executeInternal(VitessStatement.java:433) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.execute(Executor.java:132) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.newExecute(Executor.java:484) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.insideTransaction(Executor.java:689) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.lambda$executePlan$0(Executor.java:340) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.engine.InsertEngine.execute(InsertEngine.java:212) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.engine.InsertEngine.execInsertSharded(InsertEngine.java:310) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.VcursorImpl.executeMultiShard(VcursorImpl.java:100) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.Executor.executeMultiShard(Executor.java:202) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.srvtopo.ScatterConn.executeMultiShard(ScatterConn.java:218) ~[vtdriver-1.0.0.jar:?]
at com.jd.jdbc.concurrency.AllErrorRecorder.throwException(AllErrorRecorder.java:86) ~[vtdriver-1.0.0.jar:?]

Need a CI/CD enviroment based on Github Actions

Github Actions could be used to build an auto-test enviroment.

升级jetcd版本

测试过程中发现watch时报错，Network closed for unknown reason
etcd-io/jetcd#327

Optimize jetcd initialization invoke

If one node in the etcd cluster fails, application server startup is not affected (automatic fault retry)
An etcd domain name error throws an exception

The same configuration is used for read-write separation

StreamExecute does not cancel when an exception is caught during multi-shard execution

Solution

Add ctx.cancel when streamExecute catched Exception

HealthCheck Unit Test

vtdriver-spring-boot-starter

support cross-shards avg

Feature Description
If there is a cross shards SQL:

SELECT user.name, avg(user.age) FROM user GROUP BY user.name

we can modify it into Route as:

SELECT user.name, sum(user.age), count(user.age) FROM user GROUP BY user.name ORDER BY user.name

And the OrderedAggregate can compute avg value by utilizing the ordered result from Route, which is similar to the distinct_count.

Use Case(s)
SELECT user.name, avg(user.age) FROM user GROUP BY user.name

The default corePoolSize of VtQueryExecutorService may need to be adjusted

1. There can be better adjustment methods. According to the configuration of different machines, the underlying concurrency and QPS can be improved.

2. The maximumPoolSize may be smaller than corePoolSIze when initializing the thread pool

Optimized TopologyWatcher

Optimized TopologyWatcher logic to speed up the initialization time of multiple keyspace scenarios under the same Cell

Planner to move to Operator model

vitessio/vitess#11626

incorrect logic plan structure

The following complicated sql will produced an incorrected query plan:

SELECT t1.id, t2.id 
FROM t LEFT JOIN t1 on t.id = t1.idLEFT JOIN t2 on t.id = t2.id
WHERE (t2.uuid=1 or t2.uuid=2)        
     AND t.status = 'Y'        
     AND t2.status = 'Y'        
     AND t1.id in (           SELECT id FROM t4 WHERE t4.uuid=1       )       
     AND t.name='name'ORDER BY t1.id

And driver also will print following error message:

java.sql.SQLException: BUG: route is an atomic node.
at com.jd.jdbc.planbuilder.RoutePlan.supplyVar(RoutePlan.java:389)
at com.jd.jdbc.planbuilder.Jointab.procure(Jointab.java:79)
at com.jd.jdbc.sqlparser.dialect.mysql.visitor.VtRouteWireupVarFormatVisitor.format(VtRouteWireupVarFormatVisitor.java:46)
at com.jd.jdbc.sqlparser.dialect.mysql.visitor.VtRouteWireupVarFormatVisitor.visit(VtRouteWireupVarFormatVisitor.java:33)
at com.jd.jdbc.sqlparser.ast.expr.SQLIdentifierExpr.accept0(SQLIdentifierExpr.java:94)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:47)
at com.jd.jdbc.sqlparser.ast.statement.SQLSelectItem.accept0(SQLSelectItem.java:118)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:47)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:38)
at com.jd.jdbc.sqlparser.dialect.mysql.ast.statement.MySqlSelectQueryBlock.accept0(MySqlSelectQueryBlock.java:223)
at com.jd.jdbc.sqlparser.dialect.mysql.ast.statement.MySqlSelectQueryBlock.accept0(MySqlSelectQueryBlock.java:213)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:47)
at com.jd.jdbc.sqlparser.ast.statement.SQLSelect.accept0(SQLSelect.java:92)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:47)
at com.jd.jdbc.sqlparser.ast.statement.SQLSubqueryTableSource.accept0(SQLSubqueryTableSource.java:59)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:47)
at com.jd.jdbc.sqlparser.ast.statement.SQLJoinTableSource.accept0(SQLJoinTableSource.java:44)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:47)
at com.jd.jdbc.sqlparser.dialect.mysql.ast.statement.MySqlSelectQueryBlock.accept0(MySqlSelectQueryBlock.java:225)
at com.jd.jdbc.sqlparser.dialect.mysql.ast.statement.MySqlSelectQueryBlock.accept0(MySqlSelectQueryBlock.java:213)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:47)
at com.jd.jdbc.sqlparser.ast.statement.SQLUnionQuery.accept0(SQLUnionQuery.java:107)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.statement.SQLUnionQueryTableSource.accept0(SQLUnionQueryTableSource.java:45)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.acceptChild(SQLObjectImpl.java:47)
at com.jd.jdbc.sqlparser.dialect.mysql.ast.statement.MySqlSelectQueryBlock.accept0(MySqlSelectQueryBlock.java:225)
at com.jd.jdbc.sqlparser.dialect.mysql.ast.statement.MySqlSelectQueryBlock.accept0(MySqlSelectQueryBlock.java:213)
at com.jd.jdbc.sqlparser.ast.SQLObjectImpl.accept(SQLObjectImpl.java:25)
at com.jd.jdbc.planbuilder.RoutePlan.wireup(RoutePlan.java:365)
at com.jd.jdbc.planbuilder.PlanBuilder.buildSelectPlan(PlanBuilder.java:154)
at com.jd.jdbc.planbuilder.PlanBuilder.createInstructionFor(PlanBuilder.java:108)
at com.jd.jdbc.planbuilder.PlanBuilder.buildFromStmt(PlanBuilder.java:88)
at com.jd.jdbc.Executor.getPlan(Executor.java:244)
at com.jd.jdbc.Executor.newExecute(Executor.java:398)
at com.jd.jdbc.Executor.execute(Executor.java:107)
at com.jd.jdbc.vitess.VitessStatement.executeInternal(VitessStatement.java:389)
at com.jd.jdbc.vitess.VitessPreparedStatement.execute(VitessPreparedStatement.java:395)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:59)
at com.sun.proxy.$Proxy164.execute(Unknown Source)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:64)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:79)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:63)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:83)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
at sun.reflect.GeneratedMethodAccessor151.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
at com.sun.proxy.$Proxy124.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:223)
at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:57)
at com.sun.proxy.$Proxy127.queryAllVenderListByActivityPoolId(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy128.queryAllVenderListByActivityPoolId(Unknown Source)

Implementation of Vitess Gen4 Planner

Description

Vitess Gen4 Planner is a new query planner designed to enhance the query optimization capabilities in the Vitess database system.

vitessio/vitess#7280

Asynchronously initialize the apiServer/MonitorServer to reduce the time-consuming to obtain the connection for the first time

1. ApiServer and MonitorServer need to be initialized when the application obtains connection for the first time. This process is time-consuming and they should be started asynchronously to reduce connection acquisition time.

2. ApiServer does not support multiple keyspace. the subsequent keyspace could not be registered.

Performance Optimization for Streaming Queries

Description

This issue focuses on enhancing the performance of streaming queries to improve overall system efficiency. By implementing these optimizations, we aim to reduce latency, increase throughput, and improve the responsiveness of streaming queries.

not supported ResultSetMetaData.isCaseSensitive

java.sql.SQLFeatureNotSupportedException
at com.jd.jdbc.vitess.AbstractVitessResultSetMetaData.isCaseSensitive(AbstractVitessResultSetMetaData.java:30)

replace BindVariable generated by proto

replace BindVariable generated by proto

Currently, the bindvariable in vtdriver is stored by Query.BindVariable generated by proto.

However, it is not necessary for vtdriver to use it because sqls are not sent to the VTTablet throw the grpc protocol.

Maybe we can replace it by a native bindvariable class to make our project lighter.