sinkManage should not retry on sink error infinitely.

[asddongmen]

sinkManage should not retry on sink error infinitely.

[mayjiang0203]

Agree with that, but I wonder if it is possible to show the use of the sink error message in changefeed status, and also change normal to such as "normal with errors".

Hit this failure in ticdc log.

[2023/06/27 13:47:19.780 +08:00] [WARN] [manager.go:270] ["Sink manager backend sink fails"] [namespace=default] [changefeed=dr-autosyn-changefeed] [error="[CDC:ErrMySQLTxnError]MySQL txn error: Error 1146: Table 'ecasdb.dpl_event' doesn't exist"] [errorVerbose="[CDC:ErrMySQLTxnError]MySQL txn error: Error 1146: Table 'ecasdb.dpl_event' doesn't exist\[ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/[email protected]/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/errors.WrapError\n\tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).multiStmtExecute\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:631\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:703\ngithub.com/pingcap/tiflow/cdc/sinkv2/metrics.(*Statistics).RecordBatchExecution\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/metrics/statistics.go:74\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:691\ngithub.com/pingcap/tiflow/pkg/retry.run\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:680\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).Flush\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:171\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).doFlush\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:179\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).runLoop\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:135\ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn.newSink.func1\n\tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/txn_sink.go:106\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:75\nruntime.goexit\n\truntime/asm_amd64.s:1594](http://ngithub.com/pingcap/errors.AddStack/n/tgithub.com/pingcap/[email protected]/errors.go:174/ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs/n/tgithub.com/pingcap/[email protected]/normalize.go:164/ngithub.com/pingcap/tiflow/pkg/errors.WrapError/n/tgithub.com/pingcap/tiflow/pkg/errors/helper.go:34/ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).multiStmtExecute/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:631/ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1.3/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:703/ngithub.com/pingcap/tiflow/cdc/sinkv2/metrics.(*Statistics).RecordBatchExecution/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/metrics/statistics.go:74/ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries.func1/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:691/ngithub.com/pingcap/tiflow/pkg/retry.run/n/tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:57/ngithub.com/pingcap/tiflow/pkg/retry.Do/n/tgithub.com/pingcap/tiflow/pkg/retry/retry_with_opt.go:34/ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).execDMLWithMaxRetries/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:680/ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql.(*mysqlBackend).Flush/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/mysql/mysql.go:171/ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).doFlush/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:179/ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn.(*worker).runLoop/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/worker.go:135/ngithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn.newSink.func1/n/tgithub.com/pingcap/tiflow/cdc/sinkv2/eventsink/txn/txn_sink.go:106/ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/tgolang.org/x/[email protected]/errgroup/errgroup.go:75/nruntime.goexit/n/truntime/asm_amd64.s:1594)"]

while changefeed still show it is normal.

[root@tiup-0 ~]# tiup cdc:v6.5.3 cli changefeed list  --server=http://ticdc1-peer.e2e-dr-auto-sync-5r-pref-tps-1809605-1-603:8300/
Starting component `cdc`: /root/.tiup/components/cdc/v6.5.3/cdc /root/.tiup/components/cdc/v6.5.3/cdc cli changefeed list --server=http://ticdc1-peer.e2e-dr-auto-sync-5r-pref-tps-1809605-1-603:8300/
[
  {
    "id": "dr-autosyn-changefeed",
    "namespace": "default",
    "summary": {
      "state": "normal",
      "tso": 442446779705196803,
      "checkpoint": "2023-06-27 01:28:40.726",
      "error": null
    }
  }
]