tikv/migration

Migration tools for TiKV, e.g. online bulk load.

Website | Documentation | Community Chat

TiKV is an open-source, distributed, and transactional key-value database. Unlike other traditional NoSQL systems, TiKV not only provides classical key-value APIs, but also transactional APIs with ACID compliance. Built in Rust and powered by Raft, TiKV was originally created by PingCAP to complement TiDB, a distributed HTAP database compatible with the MySQL protocol.

The design of TiKV ('Ti' stands for titanium) is inspired by some great distributed systems from Google, such as BigTable, Spanner, and Percolator, and some of the latest achievements in academia in recent years, such as the Raft consensus algorithm.

If you're interested in contributing to TiKV, or want to build it from source, see CONTRIBUTING.md.

TiKV is a graduated project of the Cloud Native Computing Foundation (CNCF). If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF. For details about who's involved and how TiKV plays a role, read the CNCF announcement.

---

With the implementation of the Raft consensus algorithm in Rust and consensus state stored in RocksDB, TiKV guarantees data consistency. Placement Driver (PD), which is introduced to implement auto-sharding, enables automatic data migration. The transaction model is similar to Google's Percolator with some performance improvements. TiKV also provides snapshot isolation (SI), snapshot isolation with lock (SQL: SELECT ... FOR UPDATE), and externally consistent reads and writes in distributed transactions.

TiKV has the following key features:

• Geo-Replication

  TiKV uses Raft and the Placement Driver to support Geo-Replication.

• Horizontal scalability

  With PD and carefully designed Raft groups, TiKV excels in horizontal scalability and can easily scale to 100+ TBs of data.

• Consistent distributed transactions

  Similar to Google's Spanner, TiKV supports externally-consistent distributed transactions.

• Coprocessor support

  Similar to HBase, TiKV implements a coprocessor framework to support distributed computing.

• Cooperates with TiDB

  Thanks to the internal optimization, TiKV and TiDB can work together to be a compelling database solution with high horizontal scalability, externally-consistent transactions, support for RDBMS, and NoSQL design patterns.

Governance

See Governance.

Documentation

For instructions on deployment, configuration, and maintenance of TiKV,see TiKV documentation on our website. For more details on concepts and designs behind TiKV, see Deep Dive TiKV.

Note:

We have migrated our documentation from the TiKV's wiki page to the official website. The original Wiki page is discontinued. If you have any suggestions or issues regarding documentation, offer your feedback here.

TiKV adopters

You can view the list of TiKV Adopters.

TiKV software stack

• Placement Driver: PD is the cluster manager of TiKV, which periodically checks replication constraints to balance load and data automatically.
• Store: There is a RocksDB within each Store and it stores data into the local disk.
• Region: Region is the basic unit of Key-Value data movement. Each Region is replicated to multiple Nodes. These multiple replicas form a Raft group.
• Node: A physical node in the cluster. Within each node, there are one or more Stores. Within each Store, there are many Regions.

When a node starts, the metadata of the Node, Store and Region are recorded into PD. The status of each Region and Store is reported to PD regularly.

Quick start

Deploy a playground with TiUP

The most quickest to try out TiKV with TiDB is using TiUP, a component manager for TiDB.

You can see this page for a step by step tutorial.

Deploy a playground with binary

TiKV is able to run separately with PD, which is the minimal deployment required.

1. Download and extract binaries.

$ export TIKV_VERSION=v7.5.0
$ export GOOS=darwin  # only {darwin, linux} are supported
$ export GOARCH=amd64 # only {amd64, arm64} are supported
$ curl -O  https://tiup-mirrors.pingcap.com/tikv-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ curl -O  https://tiup-mirrors.pingcap.com/pd-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ tar -xzf tikv-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz
$ tar -xzf pd-$TIKV_VERSION-$GOOS-$GOARCH.tar.gz

2. Start PD instance.

$ ./pd-server --name=pd --data-dir=/tmp/pd/data --client-urls="http://127.0.0.1:2379" --peer-urls="http://127.0.0.1:2380" --initial-cluster="pd=http://127.0.0.1:2380" --log-file=/tmp/pd/log/pd.log

3. Start TiKV instance.

$ ./tikv-server --pd-endpoints="127.0.0.1:2379" --addr="127.0.0.1:20160" --data-dir=/tmp/tikv/data --log-file=/tmp/tikv/log/tikv.log

4. Install TiKV Client(Python) and verify the deployment, required Python 3.5+.

$ pip3 install -i https://test.pypi.org/simple/ tikv-client

from tikv_client import RawClient

client = RawClient.connect("127.0.0.1:2379")

client.put(b'foo', b'bar')
print(client.get(b'foo')) # b'bar'

client.put(b'foo', b'baz')
print(client.get(b'foo')) # b'baz'

Deploy a cluster with TiUP

You can see this manual of production-like cluster deployment presented by @c4pt0r.

Build from source

See CONTRIBUTING.md.

Client drivers

• Go (The most stable and widely used)
• Java
• Rust
• C

If you want to try the Go client, see Go Client.

Security

Security audit

A third-party security auditing was performed by Cure53. See the full report here.

Reporting Security Vulnerabilities

To report a security vulnerability, please send an email to TiKV-security group.

See Security for the process and policy followed by the TiKV project.

Communication

Communication within the TiKV community abides by TiKV Code of Conduct. Here is an excerpt:

In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.

Social Media

• Twitter
• Blog
• Reddit
• Post questions or help answer them on Stack Overflow

Slack

Join the TiKV community on Slack - Sign up and join channels on TiKV topics that interest you.

License

TiKV is under the Apache 2.0 license. See the LICENSE file for details.

Acknowledgments

• Thanks etcd for providing some great open source tools.
• Thanks RocksDB for their powerful storage engines.
• Thanks rust-clippy. We do love the great project.

migration's Issues

Ingest: peer is not leader

Job aborted due to stage failure: Task 790 in stage 6.0 failed 4 times, most recent failure: Lost task 790.3 in stage 6.0 (TID 926, 10.36.30.17, executor 60): org.tikv.common.exception.GrpcException: message: "peer is not leader for region 8677, leader may Some(id: 8680 store_id: 7)"
not_leader {
  region_id: 8677
  leader {
    id: 8680
    store_id: 7
  }
}

 at org.tikv.common.importer.ImporterStoreClient.multiIngest(ImporterStoreClient.java:152)
 at org.tikv.common.importer.ImporterClient.ingest(ImporterClient.java:225)
 at org.tikv.common.importer.ImporterClient.write(ImporterClient.java:97)
 at org.tikv.bulkload.RawKVBulkLoader.writeAndIngest(RawKVBulkLoader.scala:164)
 at org.tikv.bulkload.RawKVBulkLoader.$anonfun$bulkLoad$5(RawKVBulkLoader.scala:110)
 at org.tikv.bulkload.RawKVBulkLoader.$anonfun$bulkLoad$5$adapted(RawKVBulkLoader.scala:109)
 at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:994)
 at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:994)
 at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2139)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
 at org.apache.spark.scheduler.Task.run(Task.scala:127)
 at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
 at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

[TiKV-BR] checksum mismatch after backup

Bug Report

1. Describe the bug

Backup report checksum failure if one tikv-server crash.

2. Minimal reproduce step (Required)

Load 2kw kv data to tikv cluster with more than 3 tikv-server nodes.
Run tikv-br backup raw
During backup running, kill one tikv-server process.
Wait for backup finish.

3. What did you see instead (Required)

backup finish with success but checksum report failure.

4. What did you expect to see? (Required)

backup finish with success and checksum succeed.

5. What is your migration tool and TiKV version? (Required)

• TiKV BR:

Use ExternalSort instead of InMemory Sort to avoid OOM

migration/online-bulk-load/src/main/scala/org/tikv/bulkload/RawKVBulkLoader.scala

Line 147 in 9809584

sortedList.sort(new Comparator[Pair[ByteString, ByteString]]() {

*: The badges in README.md is incorrect

Bug Report

1. Describe the bug

The badges in README.md is incorrect.

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

• TiKV:
• TiKV CDC:
• TiKV BR:
• TiKV Online Bulk Load:

br: Return the timestamp of backup to cooperate with TiKV-CDC

Feature Request

Is your feature request related to a problem? Please describe:

TiKV-BR is designed to initialize data before the incremental data replication of TiKV-CDC. Refer to RawKV Cross Cluster Replication (tikv/rfcs#86).

To cooperate with TiKV-CDC, the procedure is expected to be:

1. TiKV-BR backup from upstream TiKV
   1.1 Get a latest TSO (supposed to be T0).
   1.2 Set BR service safepoint to T0 and TTL as 72 hours (configurable).
   1.3 Invoke backup procedure to TiKV, flush causal_ts at the beginning to make sure that timestamp of following writes must be greater than T0.
   1.4 Return the T0 after backup finished.
2. TiKV-BR restore to downstream TiKV (It would be nice the store T0 in backup meta to remind the users agin after restore).
3. TiKV-CDC create changefeed between upstream & downstream with --start-ts T0.

Update 2022.6.29

Set T0 as TSO - safe-interval. Where safe-interval is a configurable duration defaults to 1 minute.

safe-interval is a work around to avoid missing some RawKV writes just in Raft procedure during backup, and assume that all Raw writes should be finished in 1 minute.
The cons of safe-interval is that some entries would be backup or replicated redundantly.
On a system with heavy load, the safe-interval should be configured longer.
The safe-interval would be deprecated after the read-ts of RawKV is implemented. Refer to tikv/rfcs#96.

Describe the feature you'd like:

1.1 ~ 1.4 in above description.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

cdc: add GC related procedure

Feature Request

Is your feature request related to a problem? Please describe:

The GC related procedure of TiKV-CDC is expected to be:

1. On create / resume changefeed, set TiKV-CDC's service safepoint to changefeed's start-ts when it is not zero. If failed (when the service safepoint is lower than min(all service safepoints), the creation / resume of changefeed is failed.

2. Before setting every checkpoint, set service safepoint to the checkpoint-ts. If failed (would happen when start-ts is zero that the changefeed is synchronizing from the beginning), log a warning and continue, trying to set a larger service safepoint in next round.

3. TTL of service safepoint defaults to 24 hours (configurable). Which means that a changefeed can be resumed within 24 hours. After that, the changefeed is dead. Synchronize from the beginning, or another BR backup & restore, is required.

Describe the feature you'd like:

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

support ttl argument

add table encode/decode

[TiKV-BR] checksum may mismatch when enabling ttl

Bug Report

1. Describe the bug

Some KV data with ttl expired, when doing BR( backup or restore )
The checksum will mismatch.

2. Minimal reproduce step (Required)

1. enable ttl
2. set KV with a short ttl duration
3. BR costs more time than KV ttl.

3. What did you see instead (Required)

Mismatch.

4. What did you expect to see? (Required)

Consistent.

5. What is your migration tool and TiKV version? (Required)

-TiKV: v6.1.0 nightly

Add release action for tikv-br

Feature Request

Is your feature request related to a problem? Please describe:

Describe the feature you'd like:
Add release action for tikv-br

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

add PR and Issue template

BR restore summary is not accurate.

Bug Report

1. Describe the bug

restore summary 显示的 total-ranges = 0, Result = "Nothing to restore"

2. Minimal reproduce step (Required)

run tikv-br restore command

3. What did you see instead (Required)

4. What did you expect to see? (Required)

display the accurate restore file count and result message should be good enough to user.

5. What is your migration tool and TiKV version? (Required)

• TiKV BR

[TiKV-BR] Parameter Identification ERROR and Command Prompt inaccurate when not set --dst-api-version

Bug Report

1. Describe the bug

2. Minimal reproduce step (Required)

As the screenshot.

3. What did you see instead (Required)

dst-api-version identifys to dot ".".

4. What did you expect to see? (Required)

dst-api-version only supports: V1, V1TTL, V2

5. What is your migration tool and TiKV version? (Required)

• TiKV BR: Release Version: d8c0fb1
  Git Commit Hash: d8c0fb1
  Git Branch: main
  Go Version: go1.16.5
  UTC Build Time: 2022-05-25 09:30:18
  Race Enabled: false

ping TiKV port instead of sleep 60 in CI

cdc cli command fail to detect wrong format of sink-uri

Bug Report

1. Describe the bug

The sink-uri shoudl not have the http:// prefix, but cli fail to detect or correct this automatically.

cli create changefeed successfully, but pd client is continuously failing in backgroud.

[WARN] [base_client.go:251] ["[pd] failed to get cluster id"] [url=http://http:] [error="[PD:client:ErrClientGetMember]error:rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup http on 172.16.4.1:53: no such host" target:http: status:TRANSIENT_FAILURE: error:rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup http on 172.16.4.1:53: no such host" target:http: status:TRANSIENT_FAILURE"]

2. Minimal reproduce step (Required)

bin/cdc cli changefeed create --pd http://172.16.5.32:2379 --changefeed-id feed016 --sort-engine unified --sink-uri='tikv://http://172.16.5.32:39244'

3. What did you see instead (Required)

4. What did you expect to see? (Required)

cdc cli command should detect the wrong format and fails, or correct the format automatically.

5. What is your migration tool and TiKV version? (Required)

• TiKV CDC:

support code format

[gc-worker] support current pd safepoint update interface

Feature Request

Is your feature request related to a problem? Please describe:

support current pd safepoint update interface

Describe the feature you'd like:

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Optimize: remove sortByKey

sortByKey is too slow and it is not needed.
We can just delete it.

remove duplicate key after sorting

If the input RDD contains a duplicate key, the current ingest logic will fail.
We should remove duplicate key after sorting.

CDC & BR CI failed

Bug Report

1. Describe the bug

CDC & BR CI failed (see here):

• br-gosec #68
• br-lint #70
• br-ut
• cdc-gosec
• cdc-lint
• cdc-ut

2. Minimal reproduce step (Required)

File a PR.

3. What did you see instead (Required)

All CI tasks are successful.

4. What did you expect to see? (Required)

Some are failed.

5. What is your migration tool and TiKV version? (Required)

• TiKV: N/A
• TiKV CDC: latest
• TiKV BR: latest
• TiKV Online Bulk Load: N/A

cdc: unit test clientSuite.TestClientErrNoPendingRegion is not stable

Bug Report

1. Describe the bug

clientSuite.TestClientErrNoPendingRegion is not stable.

(Upstream similar issue pingcap/tiflow#2688).

2. Minimal reproduce step (Required)

make unit_test

3. What did you see instead (Required)

All unit tests are passed.

4. What did you expect to see? (Required)

FAIL: client_test.go:2848: clientSuite.TestClientErrNoPendingRegion
client_test.go:2850:
    s.testClientErrNoPendingRegion(c)
client_test.go:329:
    c.Assert(err, check.IsNil)
... value *errors.Error = &errors.Error{code:0, codeText:"CDC:ErrReachMaxTry", message:"reach maximum try: %d", redactArgsPos:[]int(nil), cause:(*errors.fundamental)(0xc00042fcc8), args:[]interface {}{20}, file:"/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go", line:65} ("[CDC:ErrReachMaxTry]reach maximum try: 20: request id 4 is not larger than 5")
Error: 6/16 03:25:03.851 +00:00] [ERROR] [client.go:1037] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1037\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:573\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:04.301 +00:00] [ERROR] [region_worker.go:683] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:683\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:04.952 +00:00] [ERROR] [region_worker.go:664] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMITTED] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:664\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:05.256 +00:00] [ERROR] [client.go:1032] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:39921, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:573\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:11.561 +00:00] [ERROR] [client.go:1028] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1028\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:573\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:20.814 +00:00] [ERROR] [client.go:364] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:181\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:362\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:350\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:746\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:514\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:364\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:350\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:746\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:514\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:20.923 +00:00] [ERROR] [client.go:364] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:181\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:362\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:350\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:746\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:514\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:364\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:350\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:746\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:514\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:21.025 +00:00] [ERROR] [client.go:364] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:181\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:362\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:350\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:746\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:514\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:364\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:350\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:746\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:514\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:21.127 +00:00] [ERROR] [client.go:364] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 5.1.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 5.1.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/[email protected]/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:181\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:362\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:350\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:746\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:514\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:364\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:350\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:746\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:514\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
Error: 6/16 03:25:23.035 +00:00] [ERROR] [client_test.go:1714] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestIncompatibleTiKV.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1714\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 6/16 03:26:28.646 +00:00] [ERROR] [client_test.go:2937] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2937\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 6/16 03:26:29.962 +00:00] [ERROR] [client_test.go:2994] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2994\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 6/16 03:26:34.239 +00:00] [ERROR] [client_test.go:3421] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3421\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 6/16 03:26:35.274 +00:00] [ERROR] [client_test.go:3507] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3507\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 6/16 03:26:44.043 +00:00] [ERROR] [client_test.go:3326] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3326\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 6/16 03:27:02.846 +00:00] [ERROR] [client_test.go:1509] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1509\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 6/16 03:27:02.922 +00:00] [ERROR] [client_test.go:1509] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1509\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 6/16 03:27:09.898 +00:00] [ERROR] [client_test.go:1320] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamSendWithError.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1320\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
OOPS: 48 passed, 1 FAILED

5. What is your migration tool and TiKV version? (Required)

• TiKV: N/A
• TiKV CDC: master
• TiKV BR: N/A
• TiKV Online Bulk Load: N/A

Add ci golang lint and gosec check for gc-worker

Feature Request

Is your feature request related to a problem? Please describe:

Describe the feature you'd like:
Add ci golang lint and gosec check for gc-worker

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Seperate RawKV region and TiDB regions

Feature Request

Is your feature request related to a problem? Please describe:

When run tikv-cdc, the resolved-ts would be mixed if there are rawkv data and tidb data in one same region.
PD provides functionality to seperate regions with a specified key.
Please ref:
tikv/pd#3839
https://github.com/pingcap/tiflow/blob/release-5.4/pkg/pdutil/region_label.go

TiKV-CDC should also implement this.

Describe the feature you'd like:

Seperate rawkv and tidb regions.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Release action support for tikv-br/tikv-cdc/gc-worker

Feature Request

Is your feature request related to a problem? Please describe:
Release action support for tikv-br/tikv-cdc/gc-worker

Describe the feature you'd like:
Release action support for tikv-br/tikv-cdc/gc-worker

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[BR] no checksum in backup/restore process

Bug Report

1. Describe the bug

Does not do checksum in back/restore process.

2. Minimal reproduce step (Required)

run tikv-br backup raw & tikv-br restore raw

3. What did you see instead (Required)

no checksum is done during back/restore.

4. What did you expect to see? (Required)

do checksum in back/restore process and report checksum result.

5. What is your migration tool and TiKV version? (Required)

• TiKV BR:

Clean TiKV-BR command args

Bug Report

1. Describe the bug

1. There are many args in tikv-br and some of them are useless or not supported well. Please remove/hide them.
2. Some help description is not suitable, for example, tikv-br help display br is a tool for TiDB/TiKV cluster..., please check all hekp descriptions.

2. Minimal reproduce step (Required)

tikv-br help
tikv-br backup help
tikv-br restore help
tikv-br backup raw --help
tikv-br restore raw --help

3. What did you see instead (Required)

1. There are many args in tikv-br and some of them are useless or not supported well. Please remove/hide them.
2. Some help description is not suitable, for example, tikv-br help display br is a tool for TiDB/TiKV cluster..., please check all hekp descriptions.

4. What did you expect to see? (Required)

all command args should be supported well, else remove them.

5. What is your migration tool and TiKV version? (Required)

• TiKV BR:

Some unit tests for module Owner are not running

Bug Report

1. Describe the bug

Because of losing func Test(t *testing.T), some unit tests for module Owner are not running.

2. Minimal reproduce step (Required)

cd cdc/cdc/owner
go test

3. What did you see instead (Required)

Some unit tests don't run;

4. What did you expect to see? (Required)

all unit tests run;

5. What is your migration tool and TiKV version? (Required)

• TiKV CDC:

support bulk load using spark

subtask of tikv/tikv#10563

Update TiKV-BR README.md

Bug Report

1. Describe the bug

README.md in migration/br is outdated. Please update it.

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

• TiKV:
• TiKV CDC:
• TiKV BR:
• TiKV Online Bulk Load:

cdc: Add sorter back to pipeline

Bug Report

1. Describe the bug

The sorter of pipeline was removed in early cdc PR.
And that time we thought that as timestamps of entries in a region are always non-decreasing, we don't need the sorter.

But now I find that actually the processor is not processing only one region, but multiple regions at a time. And now the timestamps are not in order. This will make the resolved-ts logic wrong.
See here:

	keyspanID, resolvedTs := batch[i].keyspanID, batch[i].resolvedTs
	rawKVEntries := b.buffer[keyspanID]

	i := sort.Search(len(rawKVEntries), func(i int) bool {
		return rawKVEntries[i].CRTs > resolvedTs
	})
	if i == 0 {
		continue
	}
	state.metricTotalRows.Add(float64(i))

	err := b.Sink.EmitChangedEvents(ctx, rawKVEntries...)

If the CRTs of entries are not in order, we would not get all the expected entries that earlier than resolvedTs.

Although we can do a full search here, but it would be inefficiency. As the advancing of resolvedTs would be slow, we will scan again and again.

We can implement the sorter using merge sort, to utilize the feature of non-decreasing.

Please also note that, it's possible that we would break the order of entires timestamp in the same region later. See tikv/tikv#12020.

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

• TiKV:
• TiKV CDC:
• TiKV BR:
• TiKV Online Bulk Load:

Bulk Load Document

Upgrade to go1.18 in TiKV-BR

Feature Request

Is your feature request related to a problem? Please describe:
No problem.

Describe the feature you'd like:
Upgrade to go1.18 in TiKV-BR

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Batch export

Audit License check failed

Bug Report

1. Describe the bug

Audit License / check-license (pull_request) is failed. see https://github.com/tikv/migration/runs/5152588535?check_suite_focus=true for details

2. Minimal reproduce step (Required)

submit a pull request to trigger the action

3. What did you see instead (Required)

the check passes

4. What did you expect to see? (Required)

the check failed

5. What is your migration tool and TiKV version? (Required)

• TiKV: N/A
• TiKV CDC: master
• TiKV BR: master
• TiKV Online Bulk Load: master

[TiKV-BR] BR failed to restore when restored V1TTL data, with Error: [BR:KV:ErrKVKeyNotInRegion]key not in region

Bug Report

1. Describe the bug

BR failed to restore when restored V1TTL data, with Error: [BR:KV:ErrKVKeyNotInRegion]key not in region

2. Minimal reproduce step (Required)

1. Deploy V1+TTL Cluster v6.1.0
2. Import some data
3. Backup -dst-api-version V1TTL
4. Backup -dst-api-version V2
5. Restore V2 data on the same cluster, expect to fail
6. Restore V1TTL data on the same cluster, expect to success

3. What did you see instead (Required)

Failed to restore V1TTL data

4. What did you expect to see? (Required)

Should restore V1TTL data successfully.

5. What is your migration tool and TiKV version? (Required)

• TiKV: ./tikv-server -V
  TiKV
  Release Version: 6.1.0-alpha
  Edition: Community
  Git Commit Hash: 50c2abf87fd144a739ba7ccb70b4100f2c0d92f9
  Git Commit Branch: heads/refs/tags/v6.1.0-alpha
  UTC Build Time: 2022-05-25 14:54:48
  Rust Version: rustc 1.60.0-nightly (1e12aef3f 2022-02-13)
  Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
  Profile: dist_release

• TiKV BR: ./tikv-br --version
  Release Version: 7f90200
  Git Commit Hash: 7f90200
  Git Branch: pr103
  Go Version: go1.16.5
  UTC Build Time: 2022-05-26 01:59:28
  Race Enabled: false

TiKV BR Development Tasks

TiKV BR Development Tasks

Close: #66
RFC: tikv/rfcs#86

Tasks

• TiKV-BR
  • Fix CI error (ref: #57, find CI error here) #68 #70
  • Cherrypick pingcap/tidb#32612 #77
  • Support RawKV API V2 #75, #91
  • Support migration from RawKV API V1/TTL to V2 #75, #91
  • Add checksum #91
• TiKV
  • Write development proposal & discuss with DP team
  • Support RawKV API V2 tikv/tikv#12303
  • Support migration from RawKV API V1/TTL to V2 tikv/tikv#12303
  • Add checksum (ref [raw_checksum] (https://github.com/tikv/tikv/blob/869b953e798cabf29872fd17d526a7061437aec2/src/storage/mod.rs#L2425)) tikv/tikv#12303
  • Fix tikv/tikv#12015: tikv/tikv#12303

Reference

TiKV components related to backup & restore:

• components/backup: for backup
• components/sst_importer: for restore
• split_region interface: for pre-splitting regions before restore

Some merged PR related:

• tikv/tikv#10940
• tikv/tikv#11041
• tikv/tikv#11664

[TiKV-BR] Remove dependence to etcd, pd and tidb

Bug Report

1. Describe the bug

TiKV-BR depends on many tidb and pd packages, which cause some dependence problems. tidb/br has removed the dependence to PD. Please remove the dependence to PD in TiKV-BR and reduce dependence to TiDB also.

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

• TiKV BR:

add CI

Add SSTDataSource

subtask of tikv/tikv#11007

Initialize TiKV CDC / BR repo

Motivation

Initialize TiKV CDC / BR repo, to provide CDC feature for TiKV.
Related issue: tikv/tikv#11745
Related RFC: tikv/rfcs#86

Tasks

• CDC
  • Initialize CDC codes from https://github.com/pingcap/tiflow/tree/v5.5.0-alpha to tikv/migration/cdc.
  • Build Locally
  • Unit Test Locally (ref)
  • Integration Test Locally (tikv-tikv ONLY, ref)
  • Build in CI (ref)
  • Unit in CI (ref)
  • Integration Test in CI (tikv-tikv ONLY, ref)
• BR
  • Initialize BR codes from https://github.com/pingcap/tidb/tree/v5.5.0-alpha/br to tikv/migration/br.
  • Build Locally
  • Unit Test Locally
  • Integration Test Locally
  • Build in CI
  • Unit Test in CI
  • Integration Test in CI

BulkLoadExample: More than Int.MaxValue elements error

Exception in thread "main" java.lang.IllegalArgumentException: More than Int.MaxValue elements.
        at scala.collection.immutable.NumericRange$.check$1(NumericRange.scala:318)
        at scala.collection.immutable.NumericRange$.count(NumericRange.scala:328)
        at scala.collection.immutable.NumericRange.numRangeElements$lzycompute(NumericRange.scala:53)
        at scala.collection.immutable.NumericRange.numRangeElements(NumericRange.scala:52)
        at scala.collection.immutable.NumericRange.length(NumericRange.scala:55)
        at org.apache.spark.rdd.ParallelCollectionRDD$.slice(ParallelCollectionRDD.scala:145)
        at org.apache.spark.rdd.ParallelCollectionRDD.getPartitions(ParallelCollectionRDD.scala:97)
        at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
        at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
        at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:276)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:272)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2179)
        at org.apache.spark.rdd.RDD.count(RDD.scala:1227)
        at org.tikv.bulkload.RawKVBulkLoader.getRegionSplitPoints(RawKVBulkLoader.scala:205)
        at org.tikv.bulkload.RawKVBulkLoader.bulkLoad(RawKVBulkLoader.scala:82)
        at org.tikv.bulkload.example.BulkLoadExample$.main(BulkLoadExample.scala:82)
        at org.tikv.bulkload.example.BulkLoadExample.main(BulkLoadExample.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

TiKV-BR: support RawKV API V2 & Cross API version migration

Feature Request

Is your feature request related to a problem? Please describe:

1. BR for RawKV is an experimental feature, not for production env.
2. No backup & restore tool for RawKV API V2.
3. No migration tool between RawKV API V1/V1TTL to V2.

Describe the feature you'd like:

1. Support backup & restore for RawAPI V2.
2. Support data migration from API V1/V1TTL to V2.

Describe alternatives you've considered:

N/A.

Teachability, Documentation, Adoption, Migration Strategy:

TBD.

Bulk Load Benchmark

TiKV CDC Development Tasks

TiKV CDC Development Tasks

Related issue: tikv/tikv#11745
Related RFC: tikv/rfcs#86

Tasks

• Core data structure in ETCD
  • raw changefeed
  • raw task (a part of a changefeed, for one processor)
  • ETCD path planning
• Owner
  • New changefeed for RawKV
  • Delete table changefeed
• Scheduler
  • New scheduler for Key Span
  • Delete table scheduler
• Capture
  • Support rawkv changefeed
  • Delete support of table changefeed
• Processor
  • New processor for Key Span
• Sink
  • TiKV Sink
• Command line
  • cdc cli
• Others
  • Upgrade dependencies
• TiUP support
• Metrics
  • New Grafana board for TiKV-CDC
  • Determine metrics (ref to TiCDC metrics)
• Tests
  • Integration tests
  • E2E tests

Development Guide

1. Choose a task you are interested in.
2. File a ISSUE for the task.
3. Fix the ISSUE by PR (also refer to CONTRIBUTING guide).
4. Get the PR merged.

add test for spark bulk load

BR restore failed “ BR:Restore:ErrRestoreTableIDMismatch”

Bug Report

1. Describe the bug

2. Minimal reproduce step (Required)

data format

Key： index_:_Str001_:Str001_:_{int64(20)}
Values: random string

3. What did you see instead (Required)

[2022/05/26 14:30:47.886 +08:00] [INFO] [client.go:113] ["Restore client closed"]
[2022/05/26 14:30:47.886 +08:00] [INFO] [collector.go:202] ["units canceled"] [cancel-unit=0]
[2022/05/26 14:30:47.886 +08:00] [ERROR] [client.go:752] ["[pd] fetch pending tso requests error"] [dc-location=global] [error="[PD:client:ErrClientGetTSO]context canceled: context canceled"] [stack="[github.com/tikv/pd/client.(*client).handleDispatcher\n\t/root/go_depends/pkg/mod/github.com/tikv/[email protected]/client/client.go:752](http://github.com/tikv/pd/client.(*client).handleDispatcher/n/t/root/go_depends/pkg/mod/github.com/tikv/[email protected]/client/client.go:752)"]
[2022/05/26 14:30:47.886 +08:00] [INFO] [collector.go:68] ["Raw restore failed summary"] [total-ranges=0] [ranges-succeed=0] [ranges-failed=0] [restore-files=404]
[2022/05/26 14:30:47.886 +08:00] [INFO] [client.go:666] ["[pd] exit tso dispatcher"] [dc-location=global]
[2022/05/26 14:30:47.886 +08:00] [ERROR] [restore.go:35] ["failed to restore raw kv"] [error="table id mismatch: [BR:Restore:ErrRestoreTableIDMismatch]restore table ID mismatch"]
 [errorVerbose="[BR:Restore:ErrRestoreTableIDMismatch]restore table ID mismatch\ntable id mismatch\[ngithub.com/tikv/migration/br/pkg/restore.SortRanges\n\t/root/go-projects](http://ngithub.com/tikv/migration/br/pkg/restore.SortRanges/n/t/root/go-projects)
/migration/br/pkg/restore/range.go:60\[ngithub.com/tikv/migration/br/pkg/restore.(*RegionSplitter).Split\n\t/root/go-projects/migration/br/pkg/restore/split.go:96\ngithub.co](http://ngithub.com/tikv/migration/br/pkg/restore.(*RegionSplitter).Split/n/t/root/go-projects/migration/br/pkg/restore/split.go:96/ngithub.co)
m/tikv/migration/br/pkg/restore.SplitRanges\n\t/root/go-projects/migration/br/pkg/restore/util.go:121\[ngithub.com/tikv/migration/br/pkg/task.RunRestoreRaw\n\t/root/go](http://ngithub.com/tikv/migration/br/pkg/task.RunRestoreRaw/n/t/root/go)
-projects/migration/br/pkg/task/restore_raw.go:110\nmain.runRestoreRawCommand\n\t/root/go-projects/migration/br/cmd/br/restore.go:34\nmain.newRawRestoreCommand.func1\n\t/ho
me/huayp/go-projects/migration/br/cmd/br/restore.go:72\[ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:842\n](http://ngithub.com/spf13/cobra.(*Command).execute/n/t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:842/n)
[github.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/huay](http://github.com/spf13/cobra.(*Command).ExecuteC/n/t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:950/ngithub.com/spf13/cobra.(*Command).Execute/n/t/home/huay)
p/go_depends/pkg/mod/[github.com/spf13/[email protected]/command.go:887\nmain.main\n\t/root/go-projects/migration/br/cmd/br/main.go:56\nruntime.main\n\t/usr/local/go/src/runtime/](http://github.com/spf13/[email protected]/command.go:887/nmain.main/n/t/root/go-projects/migration/br/cmd/br/main.go:56/nruntime.main/n/t/usr/local/go/src/runtime/)
proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"] [stack="main.runRestoreRawCommand\n\t/root/go-projects/migration/br/cmd/br/restore.go:35\nmain.n
ewRawRestoreCommand.func1\n\t/root/go-projects/migration/br/cmd/br/restore.go:72\[ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go_depends/pkg/mod/github.com/spf](http://ngithub.com/spf13/cobra.(*Command).execute/n/t/root/go_depends/pkg/mod/github.com/spf)
13/[email protected]/command.go:842\[ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/hom](http://ngithub.com/spf13/cobra.(*Command).ExecuteC/n/t/hom)
e/huayp/go_depends/pkg/mod/[github.com/spf13/[email protected]/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:887\nmain.main\n\t/root/go-projects/migration/br/cmd/br/main.go:56\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225](http://github.com/spf13/[email protected]/command.go:950/ngithub.com/spf13/cobra.(*Command).Execute/n/t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:887/nmain.main/n/t/root/go-projects/migration/br/cmd/br/main.go:56/nruntime.main/n/t/usr/local/go/src/runtime/proc.go:225)"]
[2022/05/26 14:30:47.886 +08:00] [ERROR] [main.go:58] ["br failed"] [error="table id mismatch: [BR:Restore:ErrRestoreTableIDMismatch]restore table ID mismatch"] [errorVerbose="[BR:Restore:ErrRestoreTableIDMismatch]restore table ID mismatch\ntable id mismatch\[ngithub.com/tikv/migration/br/pkg/restore.SortRanges\n\t/root/go-projects/migration/br/pkg/restore/range.go:60\ngithub.com/tikv/migration/br/pkg/restore.(*RegionSplitter).Split\n\t/root/go-projects/migration/br/pkg/restore/split.go:96\ngithub.com/tikv/migration/br/pkg/restore.SplitRanges\n\t/root/go-projects/migration/br/pkg/restore/util.go:121\ngithub.com/tikv/migration/br/pkg/task.RunRestoreRaw\n\t/root/go-projects/migration/br/pkg/task/restore_raw.go:110\nmain.runRestoreRawCommand\n\t/root/go-projects/migration/br/cmd/br/restore.go:34\nmain.newRawRestoreCommand.func1\n\t/root/go-projects/migration/br/cmd/br/restore.go:72\ngithub.com/spf13/cobra.(*Command).execute\n\t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:887\nmain.main\n\t/root/go-projects/migration/br/cmd/br/main.go:56\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371](http://ngithub.com/tikv/migration/br/pkg/restore.SortRanges/n/t/root/go-projects/migration/br/pkg/restore/range.go:60/ngithub.com/tikv/migration/br/pkg/restore.(*RegionSplitter).Split/n/t/root/go-projects/migration/br/pkg/restore/split.go:96/ngithub.com/tikv/migration/br/pkg/restore.SplitRanges/n/t/root/go-projects/migration/br/pkg/restore/util.go:121/ngithub.com/tikv/migration/br/pkg/task.RunRestoreRaw/n/t/root/go-projects/migration/br/pkg/task/restore_raw.go:110/nmain.runRestoreRawCommand/n/t/root/go-projects/migration/br/cmd/br/restore.go:34/nmain.newRawRestoreCommand.func1/n/t/root/go-projects/migration/br/cmd/br/restore.go:72/ngithub.com/spf13/cobra.(*Command).execute/n/t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:842/ngithub.com/spf13/cobra.(*Command).ExecuteC/n/t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:950/ngithub.com/spf13/cobra.(*Command).Execute/n/t/root/go_depends/pkg/mod/github.com/spf13/[email protected]/command.go:887/nmain.main/n/t/root/go-projects/migration/br/cmd/br/main.go:56/nruntime.main/n/t/usr/local/go/src/runtime/proc.go:225/nruntime.goexit/n/t/usr/local/go/src/runtime/asm_amd64.s:1371)"] [stack="main.main\n\t/root/go-projects/migration/br/cmd/br/main.go:58\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

• TiKV: 6.0.1 / Git Commit Hash: 50c2abf87fd144a739ba7ccb70b4100f2c0d92f9
• TiKV CDC:
• TiKV BR: tikv-br / 7f90200
• TiKV Online Bulk Load:

Use new gc safepoint update interface

Bug Report

1. Describe the bug

Use new gc safepoint update interface

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

Use new gc safepoint update interface

5. What is your migration tool and TiKV version? (Required)

GC-Worker

[TiKV-BR] Checksum mismatch report after backup

Bug Report

1. Describe the bug

checksum mismatch occurs after backup if one tikv-server crash during backup

2. Minimal reproduce step (Required)

1. Load 2kw kv data to tikv cluster with more than 3 tikv-server nodes.
2. Run tikv-br backup raw
3. During backup running, kill one tikv-server process.
4. Wait for backup finish.

3. What did you see instead (Required)

backup finish with success but checksum report failure.

4. What did you expect to see? (Required)

backup finish with success and checksum succeed.

5. What is your migration tool and TiKV version? (Required)

• TiKV BR:

cdc: Increase test coverage

Bug Report

1. Describe the bug

The test coverage is low (totally about 59.67% for master). We should increase it.
See https://app.codecov.io/gh/tikv/migration.

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

• TiKV:
• TiKV CDC:
• TiKV BR:
• TiKV Online Bulk Load:

BulkLoad cannot work normally in case of large amount of data, all ingest failed

Bug Report

1. Describe the bug

A large number of failures in the later stage of task execution，all ingest are retry again and again, until failure.

2. Minimal reproduce step (Required)

datum: 500000000 keys, and 1KB per keys.
In the initial stage of the task, each partition can be imported normally and the import speed is within 3-5 seconds. However, in the later stage of the task, each task (partition) needs to be executed for 30-40 minute, and is in the failure and retry stage. Finally, it fails.

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

• TiKV Online Bulk Load: