Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0 Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-cvzwf-lmz0h --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "2b4219014c486f1488eb502250b1373142adef33" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-cvzwf" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git rev-list --no-walk cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=9973ed59-40ef-4a55-9c99-e0206535c3eb BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Debug info BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct _=/usr/bin/printenv POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=3 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-cvzwf GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-83b4364c57a128adeb13873e1dd44198b8eb823eaae8c042ddce73e4d3360e79 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-cvzwf xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.21.6' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build385006743=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0 bash' debug command: kubectl -n jenkins-tiflow exec -ti xtiflow-release-7-5-pull-cdc-integration-storage-test-285-gz2p0 bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8984 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: 9fe97265c5e61173 (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 17 IN A 20.205.243.166 ;; Query time: 0 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Fri Apr 26 10:56:24 UTC 2024 ;; MSG SIZE rcvd: 77 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=3&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=3&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=4&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=4&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=5&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=5&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [Makefile, cdc/api/v2/api_helpers.go, cdc/api/v2/api_helpers_mock.go, cdc/api/v2/changefeed.go, cdc/api/v2/changefeed_test.go, cdc/api/v2/model.go, cdc/api/v2/model_test.go, cdc/api/v2/unsafe.go, cdc/entry/codec.go, cdc/entry/codec_test.go, cdc/entry/mounter.go, cdc/entry/mounter_test.go, cdc/entry/schema/snapshot.go, cdc/entry/schema/snapshot_test.go, cdc/entry/schema_storage.go, cdc/entry/schema_storage_test.go, cdc/entry/schema_test_helper.go, cdc/entry/validator.go, cdc/kv/client_bench_test.go, cdc/kv/client_test.go, cdc/kv/shared_client_test.go, cdc/kv/store_op.go, cdc/model/changefeed_test.go, cdc/model/codec/codec.go, cdc/model/codec/codec_test.go, cdc/model/codec/v1/codec.go, cdc/model/codec/v1/convert.go, cdc/model/codec/v1/convert_test.go, cdc/model/mounter.go, cdc/model/owner.go, cdc/model/schema_storage.go, cdc/model/schema_storage_test.go, cdc/model/sink.go, cdc/model/sink_test.go, cdc/owner/changefeed.go, cdc/owner/changefeed_test.go, cdc/owner/ddl_manager.go, cdc/owner/ddl_manager_test.go, cdc/owner/ddl_sink.go, cdc/owner/schema.go, cdc/owner/schema_test.go, cdc/processor/sourcemanager/engine/factory/factory.go, cdc/processor/sourcemanager/engine/factory/pebble.go, cdc/processor/sourcemanager/engine/pebble/db.go, cdc/processor/sourcemanager/engine/pebble/db_test.go, cdc/processor/sourcemanager/engine/pebble/event_sorter.go, cdc/processor/sourcemanager/engine/pebble/event_sorter_test.go, cdc/puller/ddl_puller.go, cdc/puller/ddl_puller_test.go, cdc/puller/puller.go, cdc/puller/puller_test.go, cdc/server/server.go, cdc/sink/ddlsink/cloudstorage/cloud_storage_ddl_sink.go, cdc/sink/ddlsink/cloudstorage/cloud_storage_ddl_sink_test.go, cdc/sink/ddlsink/mq/mq_ddl_sink_test.go, cdc/sink/ddlsink/mq/pulsar_ddl_sink_test.go, cdc/sink/ddlsink/mysql/mysql_ddl_sink.go, cdc/sink/ddlsink/mysql/mysql_ddl_sink_test.go, cdc/sink/dmlsink/cloudstorage/cloud_storage_dml_sink_test.go, cdc/sink/dmlsink/cloudstorage/defragmenter_test.go, cdc/sink/dmlsink/cloudstorage/dml_worker_test.go, cdc/sink/dmlsink/cloudstorage/encoding_worker_test.go, cdc/sink/dmlsink/mq/dispatcher/event_router.go, cdc/sink/dmlsink/mq/dispatcher/event_router_test.go, cdc/sink/dmlsink/mq/dispatcher/partition/columns_test.go, cdc/sink/dmlsink/mq/dispatcher/partition/index_value_test.go, cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go, cdc/sink/dmlsink/txn/event.go, cdc/sink/dmlsink/txn/event_test.go, cdc/sink/dmlsink/txn/mysql/dml.go, cdc/sink/dmlsink/txn/mysql/dml_test.go, cdc/sink/dmlsink/txn/mysql/mysql.go, cdc/sink/dmlsink/txn/mysql/mysql_test.go, cmd/cdc/main.go, cmd/dm-syncer/config.go, cmd/dm-worker/main.go, cmd/filter-helper/main.go, cmd/kafka-consumer/main.go, dm/chaos/cases/db.go, dm/chaos/cases/diff.go, dm/chaos/cases/schema.go, dm/chaos/cases/stmt.go, dm/chaos/cases/task.go, dm/checker/check_test.go, dm/checker/checker.go, dm/config/subtask.go, dm/config/subtask_test.go, dm/config/task.go, dm/config/task_converters.go, dm/config/task_converters_test.go, dm/config/task_test.go, dm/ctl/common/util.go, dm/dumpling/dumpling.go, dm/dumpling/dumpling_test.go, dm/loader/checkpoint.go, dm/loader/lightning.go, dm/loader/lightning_test.go, dm/master/config.go, dm/master/config_test.go, dm/master/openapi_view.go, dm/master/server.go, dm/master/server_test.go, dm/master/shardddl/optimist.go, dm/master/shardddl/optimist_test.go, dm/pkg/binlog/event/util.go, dm/pkg/checker/binlog.go, dm/pkg/checker/binlog_test.go, dm/pkg/checker/conn_checker.go, dm/pkg/checker/lightning.go, dm/pkg/checker/lightning_test.go, dm/pkg/checker/mysql_server.go, dm/pkg/checker/mysql_server_test.go, dm/pkg/checker/onlineddl.go, dm/pkg/checker/privilege.go, dm/pkg/checker/privilege_test.go, dm/pkg/checker/table_structure.go, dm/pkg/checker/table_structure_test.go, dm/pkg/checker/utils.go, dm/pkg/checker/utils_test.go, dm/pkg/conn/baseconn.go, dm/pkg/conn/baseconn_test.go, dm/pkg/conn/basedb.go, dm/pkg/conn/db.go, dm/pkg/conn/db_test.go, dm/pkg/conn/mockdb.go, dm/pkg/conn/utils.go, dm/pkg/election/election_test.go, dm/pkg/etcdutil/etcdutil_test.go, dm/pkg/log/log.go, dm/pkg/log/log_test.go, dm/pkg/parser/common.go, dm/pkg/parser/common_test.go, dm/pkg/retry/errors.go, dm/pkg/retry/strategy_test.go, dm/pkg/schema/tracker.go, dm/pkg/schema/tracker_test.go, dm/pkg/schema/visitor.go, dm/pkg/shardddl/optimism/info.go, dm/pkg/shardddl/optimism/info_test.go, dm/pkg/shardddl/optimism/keeper.go, dm/pkg/shardddl/optimism/keeper_test.go, dm/pkg/shardddl/optimism/lock.go, dm/pkg/shardddl/optimism/lock_test.go, dm/pkg/shardddl/pessimism/info_test.go, dm/pkg/upgrade/upgrade.go, dm/pkg/utils/common.go, dm/pkg/utils/common_test.go, dm/pkg/utils/time.go, dm/pkg/utils/util.go, dm/pkg/utils/util_test.go, dm/pkg/v1dbschema/schema.go, dm/relay/file_util.go, dm/relay/file_util_test.go, dm/relay/relay.go, dm/relay/relay_test.go, dm/simulator/config/config.go, dm/simulator/sqlgen/impl.go, dm/simulator/sqlgen/impl_test.go, dm/syncer/causality.go, dm/syncer/checkpoint.go, dm/syncer/checkpoint_flush_worker.go, dm/syncer/checkpoint_test.go, dm/syncer/compactor_test.go, dm/syncer/data_validator.go, dm/syncer/data_validator_test.go, dm/syncer/dbconn/db.go, dm/syncer/dbconn/upstream_db.go, dm/syncer/dbconn/utils.go, dm/syncer/ddl.go, dm/syncer/ddl_test.go, dm/syncer/dml.go, dm/syncer/dml_test.go, dm/syncer/dml_worker_test.go, dm/syncer/error.go, dm/syncer/error_test.go, dm/syncer/expr_filter_group.go, dm/syncer/expr_filter_group_test.go, dm/syncer/filter.go, dm/syncer/filter_test.go, dm/syncer/handle_error.go, dm/syncer/job.go, dm/syncer/job_test.go, dm/syncer/online-ddl-tools/online_ddl.go, dm/syncer/opt_sharding_group.go, dm/syncer/optimist.go, dm/syncer/safe-mode/mode.go, dm/syncer/safe-mode/mode_test.go, dm/syncer/schema.go, dm/syncer/shardddl/optimist.go, dm/syncer/shardddl/optimist_test.go, dm/syncer/sharding-meta/shardmeta.go, dm/syncer/sharding_group.go, dm/syncer/sharding_group_test.go, dm/syncer/status_test.go, dm/syncer/syncer.go, dm/syncer/syncer_test.go, dm/syncer/util.go, dm/syncer/util_test.go, dm/syncer/validate_worker.go, dm/syncer/validate_worker_test.go, dm/syncer/validator_checkpoint.go, dm/syncer/validator_checkpoint_test.go, dm/syncer/validator_cond.go, dm/syncer/validator_cond_test.go, dm/tests/incompatible_ddl_changes/conf/dm-task1.yaml, dm/tests/incompatible_ddl_changes/run.sh, dm/tests/lightning_mode/run.sh, dm/tests/tls/run.sh, dm/unit/unit_test.go, dm/worker/server_test.go, dm/worker/task_checker_test.go, engine/chaos/cases/dm/case.go, engine/chaos/cases/dm/db.go, engine/executor/server.go, engine/jobmaster/dm/checkpoint/agent.go, engine/jobmaster/dm/checkpoint/agent_test.go, engine/jobmaster/dm/config/config.go, engine/jobmaster/dm/ddl_coordinator.go, engine/pkg/meta/internal/etcdkv/mock.go, engine/pkg/meta/model/config.go, engine/pkg/orm/util_test.go, examples/golang/avro-checksum-verification/go.mod, examples/golang/avro-checksum-verification/go.sum, examples/golang/avro-checksum-verification/main.go, go.mod, go.sum, pkg/applier/redo.go, pkg/applier/redo_test.go, pkg/cmd/server/server.go, pkg/cmd/server/server_test.go, pkg/config/filter.go, pkg/config/outdated/v1.go, pkg/config/replica_config.go, pkg/errorutil/util.go, pkg/errorutil/util_test.go, pkg/etcd/etcd.go, pkg/filter/expr_filter.go, pkg/filter/expr_filter_test.go, pkg/filter/filter.go, pkg/filter/filter_test.go, pkg/filter/filter_test_helper.go, pkg/filter/sql_event_filter.go, pkg/filter/utils.go, pkg/filter/utils_test.go, pkg/httputil/httputil_test.go, pkg/migrate/migrate_test.go, pkg/pdutil/api_client.go, pkg/pdutil/api_client_test.go, pkg/pdutil/utils_test.go, pkg/sink/cloudstorage/path_test.go, pkg/sink/cloudstorage/table_definition.go, pkg/sink/cloudstorage/table_definition_test.go, pkg/sink/codec/avro/avro.go, pkg/sink/codec/avro/avro_test.go, pkg/sink/codec/avro/decoder.go, pkg/sink/codec/avro/decoder_test.go, pkg/sink/codec/canal/canal_encoder_test.go, pkg/sink/codec/canal/canal_entry.go, pkg/sink/codec/canal/canal_entry_test.go, pkg/sink/codec/canal/canal_json_message.go, pkg/sink/codec/canal/canal_json_row_event_encoder_test.go, pkg/sink/codec/canal/canal_json_txn_event_encoder_test.go, pkg/sink/codec/canal/canal_test_util.go, pkg/sink/codec/common/message_test.go, pkg/sink/codec/common/verify_checksum.go, pkg/sink/codec/craft/craft_encoder_test.go, pkg/sink/codec/craft/message_decoder.go, pkg/sink/codec/craft/message_encoder.go, pkg/sink/codec/csv/csv_decoder.go, pkg/sink/codec/csv/csv_decoder_test.go, pkg/sink/codec/csv/csv_encoder_test.go, pkg/sink/codec/csv/csv_message.go, pkg/sink/codec/csv/csv_message_test.go, pkg/sink/codec/internal/batch_tester.go, pkg/sink/codec/internal/column.go, pkg/sink/codec/internal/java.go, pkg/sink/codec/maxwell/maxwell_encoder_test.go, pkg/sink/codec/maxwell/maxwell_message.go, pkg/sink/codec/maxwell/maxwell_message_test.go, pkg/sink/codec/open/open_protocol_decoder.go, pkg/sink/codec/open/open_protocol_encoder_test.go, pkg/sink/codec/open/open_protocol_message.go, pkg/sink/codec/open/open_protocol_message_test.go, pkg/sink/kafka/claimcheck/claim_check_test.go, pkg/sink/mysql/db_helper.go, pkg/spanz/span.go, pkg/spanz/span_test.go, pkg/sqlmodel/causality.go, pkg/sqlmodel/row_change.go, pkg/sqlmodel/row_change_test.go, pkg/sqlmodel/utils.go, pkg/sqlmodel/where_handle.go, pkg/sqlmodel/where_handle_test.go, pkg/txnutil/gc/gc_manager.go, pkg/txnutil/gc/gc_manager_test.go, pkg/upstream/upstream.go, pkg/util/memory.go, pkg/util/tz.go, pkg/version/check.go, pkg/version/check_test.go, tests/integration_tests/cdc/dailytest/db.go, tests/integration_tests/cdc/dailytest/job.go, tests/integration_tests/cdc/dailytest/parser.go, tests/integration_tests/resolve_lock/main.go, tests/integration_tests/util/db.go] [Pipeline] echo diff file not matched: Makefile [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-823a389-aa323aa) 199979520 bytes in 1.08 secs (184508675 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.36.6 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/.git/ .git HEAD is now at ddda40f65 Merge commit 'aa323aa99660421afc797ea115dbb30242e573cf' into HEAD POST git-upload-pack (686 bytes) From https://github.com/pingcap/tiflow = [up to date] release-7.5 -> origin/release-7.5 = [up to date] refs/pull/10970/head -> origin/pr/10970/head Warning: you are leaving 1 commit behind, not connected to any of your branches: ddda40f65 Merge commit 'aa323aa99660421afc797ea115dbb30242e573cf' into HEAD If you want to keep it by creating a new branch, this may be a good time to do so with: git branch ddda40f65 HEAD is now at 823a3899c sink(ticdc): Revert changes related to the conflict detector (#10924) ๐Ÿšง Checkouting to base SHA:823a3899cc17bf14ad7875ab0cc69092186abc4a... HEAD is now at 823a3899c sink(ticdc): Revert changes related to the conflict detector (#10924) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 823a3899cc17bf14ad7875ab0cc69092186abc4a 823a3899c sink(ticdc): Revert changes related to the conflict detector (#10924) 3c4a12d87 mysql(ticdc): fix the data race in the conflict detector (#10630) (#10642) 0d6a6613d kvclient(ticdc): close the grpc client after all goroutine exited to prevent data race and panic (#10865) (#10906) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 823a3899cc17bf14ad7875ab0cc69092186abc4a ... Merge made by the 'ort' strategy. Makefile | 2 +- cdc/api/v2/api_helpers.go | 2 +- cdc/api/v2/api_helpers_mock.go | 2 +- cdc/api/v2/changefeed.go | 2 +- cdc/api/v2/changefeed_test.go | 2 +- cdc/api/v2/model.go | 2 +- cdc/api/v2/model_test.go | 2 +- cdc/api/v2/unsafe.go | 2 +- cdc/entry/codec.go | 12 +- cdc/entry/codec_test.go | 6 +- cdc/entry/mounter.go | 38 +- cdc/entry/mounter_test.go | 60 +- cdc/entry/schema/snapshot.go | 4 +- cdc/entry/schema/snapshot_test.go | 2 +- cdc/entry/schema_storage.go | 4 +- cdc/entry/schema_storage_test.go | 24 +- cdc/entry/schema_test_helper.go | 18 +- cdc/entry/validator.go | 2 +- cdc/kv/client_bench_test.go | 2 +- cdc/kv/client_test.go | 4 +- cdc/kv/shared_client_test.go | 2 +- cdc/kv/store_op.go | 6 +- cdc/model/changefeed_test.go | 2 +- cdc/model/codec/codec.go | 2 +- cdc/model/codec/codec_test.go | 6 +- cdc/model/codec/v1/codec.go | 4 +- cdc/model/codec/v1/convert.go | 2 +- cdc/model/codec/v1/convert_test.go | 6 +- cdc/model/mounter.go | 2 +- cdc/model/owner.go | 2 +- cdc/model/schema_storage.go | 10 +- cdc/model/schema_storage_test.go | 6 +- cdc/model/sink.go | 6 +- cdc/model/sink_test.go | 6 +- cdc/owner/changefeed.go | 2 +- cdc/owner/changefeed_test.go | 2 +- cdc/owner/ddl_manager.go | 2 +- cdc/owner/ddl_manager_test.go | 2 +- cdc/owner/ddl_sink.go | 6 +- cdc/owner/schema.go | 6 +- cdc/owner/schema_test.go | 21 +- .../sourcemanager/engine/factory/factory.go | 6 +- .../sourcemanager/engine/factory/pebble.go | 39 +- cdc/processor/sourcemanager/engine/pebble/db.go | 13 +- .../sourcemanager/engine/pebble/db_test.go | 1 + .../sourcemanager/engine/pebble/event_sorter.go | 6 +- .../engine/pebble/event_sorter_test.go | 8 +- cdc/puller/ddl_puller.go | 6 +- cdc/puller/ddl_puller_test.go | 6 +- cdc/puller/puller.go | 2 +- cdc/puller/puller_test.go | 4 +- cdc/server/server.go | 2 +- .../ddlsink/cloudstorage/cloud_storage_ddl_sink.go | 2 +- .../cloudstorage/cloud_storage_ddl_sink_test.go | 6 +- cdc/sink/ddlsink/mq/mq_ddl_sink_test.go | 2 +- cdc/sink/ddlsink/mq/pulsar_ddl_sink_test.go | 2 +- cdc/sink/ddlsink/mysql/mysql_ddl_sink.go | 2 +- cdc/sink/ddlsink/mysql/mysql_ddl_sink_test.go | 4 +- .../cloudstorage/cloud_storage_dml_sink_test.go | 8 +- cdc/sink/dmlsink/cloudstorage/defragmenter_test.go | 6 +- cdc/sink/dmlsink/cloudstorage/dml_worker_test.go | 6 +- .../dmlsink/cloudstorage/encoding_worker_test.go | 6 +- cdc/sink/dmlsink/mq/dispatcher/event_router.go | 2 +- .../dmlsink/mq/dispatcher/event_router_test.go | 2 +- .../mq/dispatcher/partition/columns_test.go | 2 +- .../mq/dispatcher/partition/index_value_test.go | 2 +- .../transformer/columnselector/column_selector.go | 2 +- cdc/sink/dmlsink/txn/event.go | 2 +- cdc/sink/dmlsink/txn/event_test.go | 2 +- cdc/sink/dmlsink/txn/mysql/dml.go | 2 +- cdc/sink/dmlsink/txn/mysql/dml_test.go | 4 +- cdc/sink/dmlsink/txn/mysql/mysql.go | 8 +- cdc/sink/dmlsink/txn/mysql/mysql_test.go | 12 +- cmd/cdc/main.go | 2 +- cmd/dm-syncer/config.go | 4 +- cmd/dm-worker/main.go | 3 + cmd/filter-helper/main.go | 2 +- cmd/kafka-consumer/main.go | 2 +- dm/chaos/cases/db.go | 4 +- dm/chaos/cases/diff.go | 2 +- dm/chaos/cases/schema.go | 4 +- dm/chaos/cases/stmt.go | 4 +- dm/chaos/cases/task.go | 4 +- dm/checker/check_test.go | 4 +- dm/checker/checker.go | 50 +- dm/config/subtask.go | 8 +- dm/config/subtask_test.go | 2 +- dm/config/task.go | 8 +- dm/config/task_converters.go | 4 +- dm/config/task_converters_test.go | 2 +- dm/config/task_test.go | 4 +- dm/ctl/common/util.go | 2 +- dm/dumpling/dumpling.go | 6 +- dm/dumpling/dumpling_test.go | 6 +- dm/loader/checkpoint.go | 2 +- dm/loader/lightning.go | 46 +- dm/loader/lightning_test.go | 4 +- dm/master/config.go | 10 +- dm/master/config_test.go | 16 +- dm/master/openapi_view.go | 2 +- dm/master/server.go | 2 +- dm/master/server_test.go | 14 +- dm/master/shardddl/optimist.go | 2 +- dm/master/shardddl/optimist_test.go | 14 +- dm/pkg/binlog/event/util.go | 6 +- dm/pkg/checker/binlog.go | 5 +- dm/pkg/checker/binlog_test.go | 2 +- dm/pkg/checker/conn_checker.go | 4 +- dm/pkg/checker/lightning.go | 20 +- dm/pkg/checker/lightning_test.go | 2 +- dm/pkg/checker/mysql_server.go | 2 +- dm/pkg/checker/mysql_server_test.go | 2 +- dm/pkg/checker/onlineddl.go | 4 +- dm/pkg/checker/privilege.go | 14 +- dm/pkg/checker/privilege_test.go | 4 +- dm/pkg/checker/table_structure.go | 15 +- dm/pkg/checker/table_structure_test.go | 2 +- dm/pkg/checker/utils.go | 6 +- dm/pkg/checker/utils_test.go | 4 +- dm/pkg/conn/baseconn.go | 2 +- dm/pkg/conn/baseconn_test.go | 2 +- dm/pkg/conn/basedb.go | 4 +- dm/pkg/conn/db.go | 10 +- dm/pkg/conn/db_test.go | 8 +- dm/pkg/conn/mockdb.go | 16 +- dm/pkg/conn/utils.go | 2 +- dm/pkg/election/election_test.go | 8 +- dm/pkg/etcdutil/etcdutil_test.go | 20 +- dm/pkg/log/log.go | 4 +- dm/pkg/log/log_test.go | 4 +- dm/pkg/parser/common.go | 12 +- dm/pkg/parser/common_test.go | 4 +- dm/pkg/retry/errors.go | 2 +- dm/pkg/retry/strategy_test.go | 2 +- dm/pkg/schema/tracker.go | 43 +- dm/pkg/schema/tracker_test.go | 14 +- dm/pkg/schema/visitor.go | 4 +- dm/pkg/shardddl/optimism/info.go | 4 +- dm/pkg/shardddl/optimism/info_test.go | 12 +- dm/pkg/shardddl/optimism/keeper.go | 2 +- dm/pkg/shardddl/optimism/keeper_test.go | 6 +- dm/pkg/shardddl/optimism/lock.go | 10 +- dm/pkg/shardddl/optimism/lock_test.go | 10 +- dm/pkg/shardddl/pessimism/info_test.go | 2 +- dm/pkg/upgrade/upgrade.go | 2 +- dm/pkg/utils/common.go | 41 +- dm/pkg/utils/common_test.go | 2 +- dm/pkg/utils/time.go | 6 +- dm/pkg/utils/util.go | 2 +- dm/pkg/utils/util_test.go | 2 +- dm/pkg/v1dbschema/schema.go | 4 +- dm/relay/file_util.go | 2 +- dm/relay/file_util_test.go | 2 +- dm/relay/relay.go | 5 +- dm/relay/relay_test.go | 2 +- dm/simulator/config/config.go | 2 +- dm/simulator/sqlgen/impl.go | 10 +- dm/simulator/sqlgen/impl_test.go | 4 +- dm/syncer/causality.go | 2 +- dm/syncer/checkpoint.go | 8 +- dm/syncer/checkpoint_flush_worker.go | 2 +- dm/syncer/checkpoint_test.go | 8 +- dm/syncer/compactor_test.go | 4 +- dm/syncer/data_validator.go | 4 +- dm/syncer/data_validator_test.go | 6 +- dm/syncer/dbconn/db.go | 2 +- dm/syncer/dbconn/upstream_db.go | 6 +- dm/syncer/dbconn/utils.go | 4 +- dm/syncer/ddl.go | 58 +- dm/syncer/ddl_test.go | 10 +- dm/syncer/dml.go | 12 +- dm/syncer/dml_test.go | 16 +- dm/syncer/dml_worker_test.go | 10 +- dm/syncer/error.go | 16 +- dm/syncer/error_test.go | 2 +- dm/syncer/expr_filter_group.go | 20 +- dm/syncer/expr_filter_group_test.go | 6 +- dm/syncer/filter.go | 2 +- dm/syncer/filter_test.go | 4 +- dm/syncer/handle_error.go | 4 +- dm/syncer/job.go | 2 +- dm/syncer/job_test.go | 2 +- dm/syncer/online-ddl-tools/online_ddl.go | 10 +- dm/syncer/opt_sharding_group.go | 2 +- dm/syncer/optimist.go | 2 +- dm/syncer/safe-mode/mode.go | 2 +- dm/syncer/safe-mode/mode_test.go | 2 +- dm/syncer/schema.go | 14 +- dm/syncer/shardddl/optimist.go | 4 +- dm/syncer/shardddl/optimist_test.go | 12 +- dm/syncer/sharding-meta/shardmeta.go | 4 +- dm/syncer/sharding_group.go | 4 +- dm/syncer/sharding_group_test.go | 4 +- dm/syncer/status_test.go | 4 +- dm/syncer/syncer.go | 18 +- dm/syncer/syncer_test.go | 14 +- dm/syncer/util.go | 7 +- dm/syncer/util_test.go | 10 +- dm/syncer/validate_worker.go | 8 +- dm/syncer/validate_worker_test.go | 10 +- dm/syncer/validator_checkpoint.go | 4 +- dm/syncer/validator_checkpoint_test.go | 6 +- dm/syncer/validator_cond.go | 2 +- dm/syncer/validator_cond_test.go | 10 +- .../incompatible_ddl_changes/conf/dm-task1.yaml | 28 +- dm/tests/incompatible_ddl_changes/run.sh | 96 +- dm/tests/lightning_mode/run.sh | 2 +- dm/tests/tls/run.sh | 9 +- dm/unit/unit_test.go | 4 +- dm/worker/server_test.go | 8 +- dm/worker/task_checker_test.go | 2 +- engine/chaos/cases/dm/case.go | 4 +- engine/chaos/cases/dm/db.go | 2 +- engine/executor/server.go | 4 +- engine/jobmaster/dm/checkpoint/agent.go | 14 +- engine/jobmaster/dm/checkpoint/agent_test.go | 7 +- engine/jobmaster/dm/config/config.go | 4 +- engine/jobmaster/dm/ddl_coordinator.go | 10 +- engine/pkg/meta/internal/etcdkv/mock.go | 4 +- engine/pkg/meta/model/config.go | 6 +- engine/pkg/orm/util_test.go | 2 +- examples/golang/avro-checksum-verification/go.mod | 79 +- examples/golang/avro-checksum-verification/go.sum | 311 +++-- examples/golang/avro-checksum-verification/main.go | 23 +- go.mod | 276 ++--- go.sum | 1227 ++++++-------------- pkg/applier/redo.go | 2 +- pkg/applier/redo_test.go | 2 +- pkg/cmd/server/server.go | 2 +- pkg/cmd/server/server_test.go | 2 +- pkg/config/filter.go | 2 +- pkg/config/outdated/v1.go | 2 +- pkg/config/replica_config.go | 2 +- pkg/errorutil/util.go | 8 +- pkg/errorutil/util_test.go | 4 +- pkg/etcd/etcd.go | 4 +- pkg/filter/expr_filter.go | 24 +- pkg/filter/expr_filter_test.go | 14 +- pkg/filter/filter.go | 4 +- pkg/filter/filter_test.go | 2 +- pkg/filter/filter_test_helper.go | 18 +- pkg/filter/sql_event_filter.go | 6 +- pkg/filter/utils.go | 8 +- pkg/filter/utils_test.go | 18 +- pkg/httputil/httputil_test.go | 2 +- pkg/migrate/migrate_test.go | 4 +- pkg/pdutil/api_client.go | 6 +- pkg/pdutil/api_client_test.go | 6 +- pkg/pdutil/utils_test.go | 6 +- pkg/sink/cloudstorage/path_test.go | 6 +- pkg/sink/cloudstorage/table_definition.go | 8 +- pkg/sink/cloudstorage/table_definition_test.go | 8 +- pkg/sink/codec/avro/avro.go | 8 +- pkg/sink/codec/avro/avro_test.go | 6 +- pkg/sink/codec/avro/decoder.go | 32 +- pkg/sink/codec/avro/decoder_test.go | 15 +- pkg/sink/codec/canal/canal_encoder_test.go | 2 +- pkg/sink/codec/canal/canal_entry.go | 6 +- pkg/sink/codec/canal/canal_entry_test.go | 4 +- pkg/sink/codec/canal/canal_json_message.go | 4 +- .../canal/canal_json_row_event_encoder_test.go | 2 +- .../canal/canal_json_txn_event_encoder_test.go | 2 +- pkg/sink/codec/canal/canal_test_util.go | 4 +- pkg/sink/codec/common/message_test.go | 6 +- pkg/sink/codec/common/verify_checksum.go | 190 +++ pkg/sink/codec/craft/craft_encoder_test.go | 2 +- pkg/sink/codec/craft/message_decoder.go | 4 +- pkg/sink/codec/craft/message_encoder.go | 2 +- pkg/sink/codec/csv/csv_decoder.go | 6 +- pkg/sink/codec/csv/csv_decoder_test.go | 6 +- pkg/sink/codec/csv/csv_encoder_test.go | 6 +- pkg/sink/codec/csv/csv_message.go | 10 +- pkg/sink/codec/csv/csv_message_test.go | 10 +- pkg/sink/codec/internal/batch_tester.go | 2 +- pkg/sink/codec/internal/column.go | 2 +- pkg/sink/codec/internal/java.go | 2 +- pkg/sink/codec/maxwell/maxwell_encoder_test.go | 4 +- pkg/sink/codec/maxwell/maxwell_message.go | 4 +- pkg/sink/codec/maxwell/maxwell_message_test.go | 2 +- pkg/sink/codec/open/open_protocol_decoder.go | 4 +- pkg/sink/codec/open/open_protocol_encoder_test.go | 4 +- pkg/sink/codec/open/open_protocol_message.go | 2 +- pkg/sink/codec/open/open_protocol_message_test.go | 2 +- pkg/sink/kafka/claimcheck/claim_check_test.go | 2 +- pkg/sink/mysql/db_helper.go | 4 +- pkg/spanz/span.go | 8 +- pkg/spanz/span_test.go | 2 +- pkg/sqlmodel/causality.go | 8 +- pkg/sqlmodel/row_change.go | 4 +- pkg/sqlmodel/row_change_test.go | 13 +- pkg/sqlmodel/utils.go | 2 +- pkg/sqlmodel/where_handle.go | 6 +- pkg/sqlmodel/where_handle_test.go | 6 +- pkg/txnutil/gc/gc_manager.go | 8 +- pkg/txnutil/gc/gc_manager_test.go | 2 +- pkg/upstream/upstream.go | 2 +- pkg/util/memory.go | 2 +- pkg/util/tz.go | 31 +- pkg/version/check.go | 2 +- pkg/version/check_test.go | 2 +- tests/integration_tests/cdc/dailytest/db.go | 2 +- tests/integration_tests/cdc/dailytest/job.go | 4 +- tests/integration_tests/cdc/dailytest/parser.go | 8 +- tests/integration_tests/resolve_lock/main.go | 8 +- tests/integration_tests/util/db.go | 2 +- 305 files changed, 2043 insertions(+), 2191 deletions(-) create mode 100644 pkg/sink/codec/common/verify_checksum.go ๐Ÿงพ Pre-merged result: 4a2af7498eddb73ddf965dcfc461666a0186675b 4a2af7498 Merge commit 'aa323aa99660421afc797ea115dbb30242e573cf' into HEAD aa323aa99 fix potential typos. 13c5e9410 update โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache not saved (git/pingcap/tiflow/rev-823a389-aa323aa already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/third_party_download [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] sh + cd ../tiflow + ./scripts/download-integration-test-binaries.sh release-7.5 Download binaries... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 1175 0 --:--:-- --:--:-- --:--:-- 1205 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 991 0 --:--:-- --:--:-- --:--:-- 976 100 41 100 41 0 0 989 0 --:--:-- --:--:-- --:--:-- 976 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 1115 0 --:--:-- --:--:-- --:--:-- 1138 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 2170 0 --:--:-- --:--:-- --:--:-- 2277 >>> download tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/e406d5780b18a1f2aaf6230cde4b6403991e228d/centos7/tidb-server.tar.gz 2024-04-26 18:56:53 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/e406d5780b18a1f2aaf6230cde4b6403991e228d/centos7/tidb-server.tar.gz [511580034/511580034] -> "tmp/tidb-server.tar.gz" [1] >>> download pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/58453365285465cd90bc4472cff2bad7ce4d764b/centos7/pd-server.tar.gz 2024-04-26 18:57:08 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/58453365285465cd90bc4472cff2bad7ce4d764b/centos7/pd-server.tar.gz [173219013/173219013] -> "tmp/pd-server.tar.gz" [1] >>> download tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990/centos7/tikv-server.tar.gz 2024-04-26 18:57:39 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990/centos7/tikv-server.tar.gz [810252984/810252984] -> "tmp/tikv-server.tar.gz" [1] >>> download tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-7.5/9002cc34d3b593a718b6c5260ba18f30a45ab314/centos7/tiflash.tar.gz 2024-04-26 18:57:56 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-7.5/9002cc34d3b593a718b6c5260ba18f30a45ab314/centos7/tiflash.tar.gz [263167205/263167205] -> "tmp/tiflash.tar.gz" [1] >>> download minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2024-04-26 18:58:00 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> download go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2024-04-26 18:58:02 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> download jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2024-04-26 18:58:02 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] >>> download etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2024-04-26 18:58:02 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> download sync_diff_inspector.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-d671b084_linux-amd64.tar.gz 2024-04-26 18:58:04 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-d671b084_linux-amd64.tar.gz [79877126/79877126] -> "tmp/sync_diff_inspector.tar.gz" [1] >>> download schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2024-04-26 18:58:13 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] Download SUCCESS + ls -alh ./bin total 1.7G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 26 18:58 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 26 18:58 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 26 18:58 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 26 18:58 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 18 15:55 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 01:00 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 18 15:55 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 01:00 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 18 15:55 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 18 15:55 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M Apr 18 15:25 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M Apr 18 15:48 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 33M Apr 3 18:05 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 42M Apr 3 18:05 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M Apr 3 18:05 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 pd-recover -rwxr-xr-x 1 jenkins jenkins 102M Apr 3 18:04 pd-server -rwxr-xr-x 1 jenkins jenkins 25M Apr 3 18:05 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Sep 22 2023 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M Apr 22 17:56 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M Apr 18 15:56 tiflash -rwxr-xr-x 1 jenkins jenkins 353M Apr 25 18:18 tikv-server + make check_third_party_binary /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/sync_diff_inspector /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/sync_diff_inspector ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ./bin/ + ls -alh ./bin total 1.7G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 26 18:58 . drwxr-sr-x 3 jenkins jenkins 4.0K Apr 26 18:58 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 26 18:58 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 26 18:58 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 18 15:55 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 01:00 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 18 15:55 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 01:00 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 18 15:55 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 18 15:55 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M Apr 18 15:25 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M Apr 18 15:48 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 33M Apr 3 18:05 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 42M Apr 3 18:05 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M Apr 3 18:05 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 pd-recover -rwxr-xr-x 1 jenkins jenkins 102M Apr 3 18:04 pd-server -rwxr-xr-x 1 jenkins jenkins 25M Apr 3 18:05 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Sep 22 2023 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M Apr 22 17:56 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M Apr 18 15:56 tiflash -rwxr-xr-x 1 jenkins jenkins 353M Apr 25 18:18 tikv-server + ./bin/tidb-server -V Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 + ./bin/tikv-server -V TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure + ./bin/sync_diff_inspector --version App Name: sync_diff_inspector v2.0 Release Version: v7.4.0 Git Commit Hash: d671b0840063bc2532941f02e02e12627402844c Git Branch: heads/refs/tags/v7.4.0 UTC Build Time: 2023-09-22 03:51:56 Go Version: go1.21.1 [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K Apr 26 18:58 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 26 18:58 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g4a2af7498" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 10:58:16" -X "github.com/pingcap/tiflow/pkg/version.GitHash=4a2af7498eddb73ddf965dcfc461666a0186675b" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g4a2af7498"' -o bin/cdc ./cmd/cdc/main.go go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading golang.org/x/net v0.24.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/gin-gonic/gin v1.8.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading go.uber.org/multierr v1.11.0 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading cloud.google.com/go v0.112.0 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/stretchr/testify v1.9.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/goccy/go-json v0.10.2 go: downloading blainsmith.com/go/seahash v1.2.1 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading golang.org/x/sys v0.19.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/klauspost/compress v1.17.7 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20230619063055-fd67d94318fd go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/swaggo/swag v1.8.3 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/go-playground/validator/v10 v10.11.1 go: downloading github.com/pelletier/go-toml/v2 v2.0.5 go: downloading github.com/ugorji/go/codec v1.2.7 go: downloading google.golang.org/protobuf v1.33.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading cloud.google.com/go/compute v1.24.0 go: downloading cloud.google.com/go/iam v1.1.6 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading google.golang.org/api v0.162.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.8 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading golang.org/x/text v0.14.0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/spec v0.20.6 go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/go-playground/universal-translator v0.18.0 go: downloading github.com/leodido/go-urn v1.2.1 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.17.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading github.com/go-openapi/jsonpointer v0.19.6 go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/go-playground/locales v0.14.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading k8s.io/api v0.28.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/rivo/uniseg v0.4.6 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading k8s.io/apimachinery v0.28.6 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_kafka_consumer ']' + make kafka_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g4a2af7498" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:35" -X "github.com/pingcap/tiflow/pkg/version.GitHash=4a2af7498eddb73ddf965dcfc461666a0186675b" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g4a2af7498"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go + '[' -f ./bin/cdc_storage_consumer ']' + make storage_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g4a2af7498" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:47" -X "github.com/pingcap/tiflow/pkg/version.GitHash=4a2af7498eddb73ddf965dcfc461666a0186675b" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g4a2af7498"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g4a2af7498" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:55" -X "github.com/pingcap/tiflow/pkg/version.GitHash=4a2af7498eddb73ddf965dcfc461666a0186675b" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g4a2af7498"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g4a2af7498" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:55" -X "github.com/pingcap/tiflow/pkg/version.GitHash=4a2af7498eddb73ddf965dcfc461666a0186675b" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g4a2af7498"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g4a2af7498" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:55" -X "github.com/pingcap/tiflow/pkg/version.GitHash=4a2af7498eddb73ddf965dcfc461666a0186675b" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g4a2af7498"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20231213065948-336e064b488d go: downloading github.com/gogo/gateway v1.1.0 go: downloading gorm.io/gorm v1.23.8 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading go.uber.org/dig v1.13.0 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading github.com/glebarez/go-sqlite v1.17.3 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/glebarez/sqlite v1.4.6 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading gorm.io/driver/mysql v1.3.3 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading modernc.org/libc v1.16.8 go: downloading modernc.org/sqlite v1.17.3 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading modernc.org/memory v1.1.1 go: downloading modernc.org/mathutil v1.6.0 CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g4a2af7498" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:55" -X "github.com/pingcap/tiflow/pkg/version.GitHash=4a2af7498eddb73ddf965dcfc461666a0186675b" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g4a2af7498"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g4a2af7498" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:55" -X "github.com/pingcap/tiflow/pkg/version.GitHash=4a2af7498eddb73ddf965dcfc461666a0186675b" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g4a2af7498"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.1G drwxr-sr-x 2 jenkins jenkins 4.0K Apr 26 19:07 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 26 18:58 .. -rwxr-xr-x 1 jenkins jenkins 203M Apr 26 19:07 cdc -rwxr-xr-x 1 jenkins jenkins 338M Apr 26 19:07 cdc.test -rwxr-xr-x 1 jenkins jenkins 182M Apr 26 19:02 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 182M Apr 26 19:02 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 181M Apr 26 19:02 cdc_storage_consumer + ./bin/cdc version Release Version: v7.5.1-57-g4a2af7498 Git Commit Hash: 4a2af7498eddb73ddf965dcfc461666a0186675b Git Branch: HEAD UTC Build Time: 2024-04-26 11:01:55 Go Version: go version go1.21.6 linux/amd64 Failpoint Build: true [Pipeline] } Cache not saved (binary/pingcap/tiflow/cdc-integration-test/rev-823a389-aa323aa already exists) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/sync_diff_inspector ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ./bin/ + ls -alh ./bin total 2.7G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 26 19:07 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 26 18:58 .. drwxr-sr-x 2 jenkins jenkins 4.0K Apr 26 19:07 bin -rwxr-xr-x 1 jenkins jenkins 203M Apr 26 19:07 cdc -rwxr-xr-x 1 jenkins jenkins 338M Apr 26 19:07 cdc.test -rwxr-xr-x 1 jenkins jenkins 182M Apr 26 19:02 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 182M Apr 26 19:02 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 181M Apr 26 19:02 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K Apr 26 19:07 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 26 19:07 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 26 19:07 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 26 19:07 jq drwxr-sr-x 3 jenkins jenkins 4.0K Apr 26 19:07 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 26 19:07 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Apr 26 19:07 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 26 19:07 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Apr 26 19:07 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 26 19:07 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 26 19:07 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M Apr 26 19:07 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M Apr 26 19:07 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Apr 26 19:07 minio -rwxr-xr-x 1 jenkins jenkins 33M Apr 26 19:07 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 42M Apr 26 19:07 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M Apr 26 19:07 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 26 19:07 pd-recover -rwxr-xr-x 1 jenkins jenkins 102M Apr 26 19:07 pd-server -rwxr-xr-x 1 jenkins jenkins 25M Apr 26 19:07 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 26 19:07 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K Apr 26 19:07 share -rwxr-xr-x 1 jenkins jenkins 29M Apr 26 19:07 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Apr 26 19:07 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M Apr 26 19:07 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M Apr 26 19:07 tiflash -rwxr-xr-x 1 jenkins jenkins 353M Apr 26 19:07 tikv-server [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 99.18 secs (33929732 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13 [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110 [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9 [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80 [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5 [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf [Pipeline] podTemplate [Pipeline] { Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node [Pipeline] node Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-s9f8b-jf9nd --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "02e2822e5eee5ce70abe945a32ab5fe38ee22319" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-s9f8b" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33 Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@564ba944; decorates RemoteLauncher[hudson.remoting.Channel@71f441f9:JNLP4-connect connection from 10.233.86.192/10.233.86.192:59796] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-8rrnn-p9ct6 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ef90350c6938aa3d2640fd4f7823bfc171a1ab0b" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-8rrnn" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13โ€™ is offline [Pipeline] podTemplate [Pipeline] { [Pipeline] node No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@342ffd65; decorates RemoteLauncher[hudson.remoting.Channel@c6056c9:JNLP4-connect connection from 10.233.84.74/10.233.84.74:34822] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] node Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9โ€™ is offline > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8 [Pipeline] podTemplate Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-fx8js-hg6wg --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "2cf0dc59f1c6eb9127c35f605658e68ccc982164" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-fx8js" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" [Pipeline] { Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test [Pipeline] node [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] checkout > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@2bbac369; decorates RemoteLauncher[hudson.remoting.Channel@bb73abd:JNLP4-connect connection from 10.233.84.30/10.233.84.30:51922] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Commit message: "feat(tidb): test flashbacktest package (#2942)" Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_storage_test.yaml from git https://github.com/PingCAP-QE/ci.git > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8qโ€™ is offline Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-mt53l-28359 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "73ee8332ee008cae078670048068391b3a5f00f2" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-mt53l" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zzโ€™ is offline Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9โ€™ is offline Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80โ€™ is offline Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5โ€™ is offline Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvfโ€™ is offline Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-f000b-1pnb7 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "9af3d5e9282dcd615227dd02880f84b5f2df6719" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-f000b" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-rxbkl-gj694 Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-9jtg1-7x7kh --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" --- jenkins/label-digest: "ec24fdca605b1e311c971d839c6498981e0a5b45" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-rxbkl" apiVersion: "v1" kind: "Pod" metadata: annotations: name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf" namespace: "jenkins-tiflow" buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" jenkins/label-digest: "0a5e181e45b064c6ab1bb2a2551ed4a959f0a225" operator: "In" values: - "amd64" containers: jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-9jtg1" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true - key: "kubernetes.io/arch" operator: "In" values: - "amd64" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "16Gi" cpu: "6" requests: resources: limits: memory: "128Mi" cpu: "100m" tty: true memory: "12Gi" cpu: "4" tty: true volumeMounts: volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" resources: limits: memory: "256Mi" cpu: "100m" tty: true name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" name: "report" resources: limits: memory: "256Mi" cpu: "100m" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" - name: "JENKINS_AGENT_NAME" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8" - name: "JENKINS_NAME" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" name: "jnlp" resources: requests: memory: "256Mi" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-gwh03-w9776 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "a28cb7af7e3f04755d32c8443476ce441ea203e1" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-gwh03" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-61tk3-pm6jx Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-p1kck-qjtw2 --- apiVersion: "v1" kind: "Pod" metadata: annotations: --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "8430c3694fa923c2034434270549d05fe1725a0a" jenkins/label-digest: "c83e592ce27aefe1e507d36781fda8ee8aa695e3" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-61tk3" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-p1kck" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q" namespace: "jenkins-tiflow" spec: affinity: namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false tty: true volumeMounts: - mountPath: "/home/jenkins/agent" - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "net-tool" resources: limits: memory: "128Mi" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "report" resources: limits: memory: "256Mi" cpu: "100m" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" name: "report" resources: limits: memory: "256Mi" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms" - name: "JENKINS_NAME" value: "********" - name: "JENKINS_TUNNEL" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q" - name: "JENKINS_NAME" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: - emptyDir: medium: "" name: "workspace-volume" memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 8.47 secs (397494207 bytes/sec) [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] sh [Pipeline] node [Pipeline] node [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75โ€™ is offline [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6b9e6864; decorates RemoteLauncher[hudson.remoting.Channel@7b8f78fb:JNLP4-connect connection from 10.233.100.169/10.233.100.169:44736] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@42274c27; decorates RemoteLauncher[hudson.remoting.Channel@2e7c014a:JNLP4-connect connection from 10.233.67.56/10.233.67.56:50678] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@51c5d689; decorates RemoteLauncher[hudson.remoting.Channel@c2b1c25:JNLP4-connect connection from 10.233.67.213/10.233.67.213:53958] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@153c2634; decorates RemoteLauncher[hudson.remoting.Channel@4e78184a:JNLP4-connect connection from 10.233.100.110/10.233.100.110:57510] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@10e7f185; decorates RemoteLauncher[hudson.remoting.Channel@5addd203:JNLP4-connect connection from 10.233.100.41/10.233.100.41:59500] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@18c469c; decorates RemoteLauncher[hudson.remoting.Channel@68c93a1d:JNLP4-connect connection from 10.233.100.125/10.233.100.125:57010] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G02 Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01 Created Pod: kubernetes jenkins-tiflow/xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@670351b; decorates RemoteLauncher[hudson.remoting.Channel@283f1a79:JNLP4-connect connection from 10.233.86.147/10.233.86.147:44034] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_txn_v2 kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=a3e2d068-56f6-4fda-a6ef-bf2c1a9757e2 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G02 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-s9f8b GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-s9f8b xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:09:52 CST 2024] <<<<<< run test case consistent_replicate_ddl success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Avoid second fetch Commit message: "feat(tidb): test flashbacktest package (#2942)" Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: storage... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17C9CFF875C16BA4 < X-Xss-Protection: 1; mode=block < Date: Fri, 26 Apr 2024 11:09:59 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Fri Apr 26 19:09:59 CST 2024] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:10:02 CST 2024] <<<<<< run test case consistent_replicate_nfs success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36โ€™ is offline Still waiting to schedule task โ€˜xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01โ€™ is offline find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:10:09 CST 2024] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 18.09 secs (186065527 bytes/sec) [Pipeline] { [Pipeline] cache Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-8tl6s-f4wkl --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "7d006c42f522799676c292037ae6932e3a7d6289" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-8tl6s" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:10:13 CST 2024] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: storage... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17C9CFFCEC69EE46 < X-Xss-Protection: 1; mode=block < Date: Fri, 26 Apr 2024 11:10:18 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Fri Apr 26 19:10:18 CST 2024] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Exiting on signal: INTERRUPT Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 13.78 secs (244233428 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G03 Run cases: row_format drop_many_tables processor_stop_delay partition_table PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=1517b6cb-b742-4d43-b697-82ffde8bdda2 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G03 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-8rrnn GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-8rrnn xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G06 Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=c96c13ca-517c-4d4e-9e25-a0b6bf103222 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G06 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-fx8js GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx xtiflow_release-7_5_pull_cdc_integration_storage_test_285-fx8js GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:10:28 CST 2024] <<<<<< run test case consistent_partition_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7412bf62; decorates RemoteLauncher[hudson.remoting.Channel@7928402b:JNLP4-connect connection from 10.233.66.214/10.233.66.214:59330] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_big_txn_v2/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:10:32 CST 2024] <<<<<< run test case kafka_big_txn_v2 success! >>>>>> Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Verifying downstream PD is started... Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-spfqd-7rpmg Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-609qt-jw1xx --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c3bb67675af10b5d497f04a10d25edd8bcca3ea8" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-spfqd" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c490d6b1047d0f49e713a5066be619d38baebc29" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-609qt" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Commit message: "feat(tidb): test flashbacktest package (#2942)" find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:10:35 CST 2024] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 8.40 secs (400845134 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:10:41 CST 2024] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6866b8880017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:1147, start at 2024-04-26 19:10:41.744291971 +0800 CST m=+5.194688622 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:41.751 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:41.747 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:41.747 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6866b8880017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:1147, start at 2024-04-26 19:10:41.744291971 +0800 CST m=+5.194688622 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:41.751 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:41.747 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:41.747 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6866bac00015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:1214, start at 2024-04-26 19:10:41.889191455 +0800 CST m=+5.283798360 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:41.896 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:41.889 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:41.889 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6866b6f40012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:1190, start at 2024-04-26 19:10:41.630589725 +0800 CST m=+5.303976707 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:41.638 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:41.597 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:41.597 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6866b6f40012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:1190, start at 2024-04-26 19:10:41.630589725 +0800 CST m=+5.303976707 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:41.638 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:41.597 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:41.597 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6866b86c000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:1270, start at 2024-04-26 19:10:41.704338785 +0800 CST m=+5.325371734 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:41.712 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:41.691 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:41.691 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:10:44 CST 2024] <<<<<< run test case multi_topics_v2 success! >>>>>> + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2507.out cli tso query --pd=http://127.0.0.1:2379 Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-f22n4-k4m8m Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-bj25g-q9h43 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "87cdf662ed7d6de5c02c28b54d6866c648c49c85" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-f22n4" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false --- apiVersion: "v1" kind: "Pod" metadata: annotations: - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "077223fd995c91ac38e7afc25d3c9074b892e913" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-bj25g" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5" namespace: "jenkins-tiflow" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" - emptyDir: medium: "" name: "workspace-volume" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test + set +x + tso='449348854126477313 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348854126477313 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:10:46 CST 2024] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.25392541.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2510.out cli tso query --pd=http://127.0.0.1:2379 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=storage + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run storage + '[' storage '!=' storage ']' + rm -rf /tmp/tidb_cdc_test/storage_cleanup + mkdir -p /tmp/tidb_cdc_test/storage_cleanup + start_tidb_cluster --workdir /tmp/tidb_cdc_test/storage_cleanup The 1 times to try to start tidb cluster... + set +x + tso='449348854611443713 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348854611443713 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "threadcount"="2" "updateproportion"="0" "dotransactions"="false" "readallfields"="true" "mysql.host"="127.0.0.1" "mysql.db"="sink_retry" "mysql.port"="4000" "mysql.user"="root" "scanproportion"="0" "operationcount"="0" "insertproportion"="0" "requestdistribution"="uniform" "workload"="core" "recordcount"="10" "readproportion"="0" ********************************************** Run finished, takes 11.362769ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1776.3, Avg(us): 2184, Min(us): 1066, Max(us): 5709, 95th(us): 6000, 99th(us): 6000 [Fri Apr 26 19:10:49 CST 2024] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.25672569.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:10:49 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4df528a6-96f1-4759-a120-14d6159421d1 {"id":"4df528a6-96f1-4759-a120-14d6159421d1","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a198611f4 4df528a6-96f1-4759-a120-14d6159421d1 /tidb/cdc/default/default/upstream/7362131574562744203 {"id":7362131574562744203,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4df528a6-96f1-4759-a120-14d6159421d1 {"id":"4df528a6-96f1-4759-a120-14d6159421d1","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a198611f4 4df528a6-96f1-4759-a120-14d6159421d1 /tidb/cdc/default/default/upstream/7362131574562744203 {"id":7362131574562744203,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4df528a6-96f1-4759-a120-14d6159421d1 {"id":"4df528a6-96f1-4759-a120-14d6159421d1","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a198611f4 4df528a6-96f1-4759-a120-14d6159421d1 /tidb/cdc/default/default/upstream/7362131574562744203 {"id":7362131574562744203,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2586.out cli changefeed create --start-ts=449348854126477313 '--sink-uri=file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-11814?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 69e8ef5a-f6f0-44fa-87bb-3176094e930e Info: {"upstream_id":7362131574562744203,"namespace":"default","id":"69e8ef5a-f6f0-44fa-87bb-3176094e930e","sink_uri":"file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-11814?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:10:50.101132111+08:00","start_ts":449348854126477313,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348854126477313,"checkpoint_ts":449348854126477313,"checkpoint_time":"2024-04-26 19:10:45.148"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-32ksl-v75zg --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "d4f2c9a4c00b1f3423824169b741e84f309eb440" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-32ksl" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test + set +x + workdir=/tmp/tidb_cdc_test/row_format + sink_uri='file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-11814?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/row_format ++ date + echo '[Fri Apr 26 19:10:51 CST 2024] <<<<<< START storage consumer in row_format case >>>>>>' [Fri Apr 26 19:10:51 CST 2024] <<<<<< START storage consumer in row_format case >>>>>> + cd /tmp/tidb_cdc_test/row_format + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/row_format + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/row_format/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/row_format/storage_test/ticdc-row-format-test-11814?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-x64xg-tqxjx --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "08ca36d559fa1d43c74f24cd0728ebcfeb3622e5" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-x64xg" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" start tidb cluster in /tmp/tidb_cdc_test/storage_cleanup Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:10:52 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/94e1a402-17b8-4e2c-a412-4aef5a2c5ff4 {"id":"94e1a402-17b8-4e2c-a412-4aef5a2c5ff4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19870b0a 94e1a402-17b8-4e2c-a412-4aef5a2c5ff4 /tidb/cdc/default/default/upstream/7362131569678795897 {"id":7362131569678795897,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/94e1a402-17b8-4e2c-a412-4aef5a2c5ff4 {"id":"94e1a402-17b8-4e2c-a412-4aef5a2c5ff4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19870b0a 94e1a402-17b8-4e2c-a412-4aef5a2c5ff4 /tidb/cdc/default/default/upstream/7362131569678795897 {"id":7362131569678795897,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/94e1a402-17b8-4e2c-a412-4aef5a2c5ff4 {"id":"94e1a402-17b8-4e2c-a412-4aef5a2c5ff4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19870b0a 94e1a402-17b8-4e2c-a412-4aef5a2c5ff4 /tidb/cdc/default/default/upstream/7362131569678795897 {"id":7362131569678795897,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2614.out cli changefeed create --start-ts=449348854611443713 '--sink-uri=file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-1592?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: c5bf02f3-5f4f-4fe0-be2f-6d47c21b52f2 Info: {"upstream_id":7362131569678795897,"namespace":"default","id":"c5bf02f3-5f4f-4fe0-be2f-6d47c21b52f2","sink_uri":"file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-1592?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:10:53.00368039+08:00","start_ts":449348854611443713,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348854611443713,"checkpoint_ts":449348854611443713,"checkpoint_time":"2024-04-26 19:10:46.998"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + workdir=/tmp/tidb_cdc_test/sink_retry + sink_uri='file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-1592?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/sink_retry ++ date + echo '[Fri Apr 26 19:10:54 CST 2024] <<<<<< START storage consumer in sink_retry case >>>>>>' [Fri Apr 26 19:10:54 CST 2024] <<<<<< START storage consumer in sink_retry case >>>>>> + cd /tmp/tidb_cdc_test/sink_retry + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/sink_retry + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/sink_retry/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/sink_retry/storage_test/ticdc-sink-retry-test-1592?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 18.02 secs (186728090 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table row_format.finish_mark not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686807ec0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:2137, start at 2024-04-26 19:11:03.169595258 +0800 CST m=+5.702123826 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:03.176 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:03.163 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:03.163 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686807ec0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:2137, start at 2024-04-26 19:11:03.169595258 +0800 CST m=+5.702123826 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:03.176 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:03.163 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:03.163 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686808e00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:2205, start at 2024-04-26 19:11:03.232238515 +0800 CST m=+5.688696266 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:03.240 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:03.224 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:03.224 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/storage_cleanup/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/storage_cleanup/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/storage_cleanup/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table row_format.finish_mark not exists for 4-th check, retry later + cd /tmp/tidb_cdc_test/storage_cleanup + run_cdc_server --workdir /tmp/tidb_cdc_test/storage_cleanup --binary cdc.test [Fri Apr 26 19:11:08 CST 2024] <<<<<< START cdc server in storage_cleanup case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.35613563.out server --log-file /tmp/tidb_cdc_test/storage_cleanup/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/storage_cleanup/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table row_format.finish_mark not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > table row_format.finish_mark not exists for 6-th check, retry later < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:11:11 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/68bf98e1-c652-44e1-8700-d3cd94302449 {"id":"68bf98e1-c652-44e1-8700-d3cd94302449","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19d9b6f1 68bf98e1-c652-44e1-8700-d3cd94302449 /tidb/cdc/default/default/upstream/7362131664553394158 {"id":7362131664553394158,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/68bf98e1-c652-44e1-8700-d3cd94302449 {"id":"68bf98e1-c652-44e1-8700-d3cd94302449","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19d9b6f1 68bf98e1-c652-44e1-8700-d3cd94302449 /tidb/cdc/default/default/upstream/7362131664553394158 {"id":7362131664553394158,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/68bf98e1-c652-44e1-8700-d3cd94302449 {"id":"68bf98e1-c652-44e1-8700-d3cd94302449","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19d9b6f1 68bf98e1-c652-44e1-8700-d3cd94302449 /tidb/cdc/default/default/upstream/7362131664553394158 {"id":7362131664553394158,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + generate_historic_files + local target_bucket=storage_test ++ date -d yesterday +%Y-%m-%d + yesterday=2024-04-25 ++ date -d '2 days ago' +%Y-%m-%d + day_before_yesterday=2024-04-24 + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_data_type 2024-04-25 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_data_type + local day=2024-04-25 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_charset 2024-04-24 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_charset + local day=2024-04-24 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test binary_columns 2024-04-24 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=binary_columns + local day=2024-04-24 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_data_type_dummy 2024-04-24 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_data_type_dummy + local day=2024-04-24 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test multi_charset_dummy 2024-04-24 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=multi_charset_dummy + local day=2024-04-24 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test binary_columns_dummy 2024-04-25 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test + local table=binary_columns_dummy + local day=2024-04-25 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 multi_data_type 2024-04-24 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=multi_data_type + local day=2024-04-24 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 multi_charset 2024-04-24 10 true + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=multi_charset + local day=2024-04-24 + local file_cnt=10 + local should_clean=true + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/1.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/2.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/3.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/4.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/5.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/6.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/7.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/8.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/9.data + '[' true == true ']' + CLEANED_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/10.data + '[' true == true ']' + CLEANED_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test test2 binary_columns 2024-04-25 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test + local schema=test2 + local table=binary_columns + local day=2024-04-25 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test multi_data_type 2022-01-01 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=multi_data_type + local day=2022-01-01 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test multi_charset 2022-01-02 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=multi_charset + local day=2022-01-02 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/meta/CDC.index + generate_single_table_files /tmp/tidb_cdc_test/storage_cleanup storage_test_default test binary_columns 2022-01-03 10 false + local workdir=/tmp/tidb_cdc_test/storage_cleanup + local bucket=storage_test_default + local schema=test + local table=binary_columns + local day=2022-01-03 + local file_cnt=10 + local should_clean=false + table_dir=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03 + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03 ++ seq 1 10 + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data + '[' false == true ']' + EXIST_FILES+=($file) + for i in '$(seq 1 $file_cnt)' + file=/tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data + '[' false == true ']' + EXIST_FILES+=($file) + mkdir -p /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/meta + touch /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/meta/CDC.index + SINK_URI_DEFAULT='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' + run_cdc_cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' -c default-config-test --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed-default.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.cli.3785.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s' -c default-config-test --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed-default.toml Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 7.46 secs (451002555 bytes/sec) [Pipeline] { [Pipeline] cache Create changefeed successfully! ID: default-config-test Info: {"upstream_id":7362131664553394158,"namespace":"default","id":"default-config-test","sink_uri":"file:///tmp/tidb_cdc_test/storage_cleanup/storage_test_default?flush-interval=5s","create_time":"2024-04-26T19:11:11.762161703+08:00","start_ts":449348861064118278,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348861064118278,"checkpoint_ts":449348861064118278,"checkpoint_time":"2024-04-26 19:11:11.613"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table row_format.finish_mark not exists for 7-th check, retry later + set +x + sleep 20 table row_format.finish_mark not exists for 8-th check, retry later Agent xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01 is provisioned from template xtiflow_release-7_5_pull_cdc_integration_storage_test_285-sl7qz-p4vwq --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "b503d8439fc6bf00ce961bc1e77ee356b775871a" jenkins/label: "xtiflow_release-7_5_pull_cdc_integration_storage_test_285-sl7qz" name: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01" - name: "JENKINS_NAME" value: "xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test table row_format.finish_mark not exists for 9-th check, retry later table row_format.finish_mark not exists for 10-th check, retry later table row_format.finish_mark not exists for 11-th check, retry later table row_format.finish_mark not exists for 12-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 8.99 secs (374441158 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark not exists for 13-th check, retry later table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "updateproportion"="0" "readallfields"="true" "workload"="core" "mysql.user"="root" "dotransactions"="false" "operationcount"="0" "scanproportion"="0" "recordcount"="10" "insertproportion"="0" "mysql.port"="4000" "mysql.db"="sink_retry" "threadcount"="2" "readproportion"="0" ********************************************** Run finished, takes 4.522107ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3261.0, Avg(us): 805, Min(us): 499, Max(us): 1794, 95th(us): 2000, 99th(us): 2000 table row_format.finish_mark not exists for 14-th check, retry later table row_format.finish_mark not exists for 15-th check, retry later table row_format.finish_mark not exists for 16-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 7.41 secs (454358613 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark not exists for 17-th check, retry later table row_format.finish_mark not exists for 18-th check, retry later + check_file_exists true + local all_should_exist=true + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' true == true ']' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/10.data ']' + SINK_URI='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' + run_cdc_cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_cleanup.cli.3855.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml Create changefeed successfully! ID: eae28a56-ea4b-4efb-9207-2a9f1d7a5a69 Info: {"upstream_id":7362131664553394158,"namespace":"default","id":"eae28a56-ea4b-4efb-9207-2a9f1d7a5a69","sink_uri":"file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s","create_time":"2024-04-26T19:11:33.659826976+08:00","start_ts":449348866817916932,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"cloud_storage_config":{"file_expiration_days":1,"file_cleanup_cron_spec":"* * * * * *"},"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348866817916932,"checkpoint_ts":449348866817916932,"checkpoint_time":"2024-04-26 19:11:33.562"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 table row_format.finish_mark not exists for 19-th check, retry later table row_format.finish_mark not exists for 20-th check, retry later table row_format.finish_mark not exists for 21-th check, retry later table row_format.finish_mark not exists for 22-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 9.31 secs (361348124 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark not exists for 23-th check, retry later table row_format.finish_mark not exists for 24-th check, retry later table row_format.finish_mark not exists for 25-th check, retry later table row_format.finish_mark not exists for 26-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 5.65 secs (596034443 bytes/sec) [Pipeline] { [Pipeline] sh table row_format.finish_mark not exists for 27-th check, retry later [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G01 Run cases: http_api http_api_tls api_v2 canal_json_adapter_compatibility canal_json_basic multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=2401414b-cf59-4f17-8ef2-9e41b5d21e34 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G01 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-mt53l GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-mt53l xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:11:53 CST 2024] <<<<<< run test case http_api success! >>>>>> [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G11 Run cases: resolve_lock move_table autorandom generate_column PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=9157ab8d-8d3d-43fe-9369-71f1ff560993 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G11 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-f000b GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33 xtiflow_release-7_5_pull_cdc_integration_storage_test_285-f000b GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: storage... <<================= [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G10 Run cases: default_value simple cdc_server_tips event_filter PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=748a4b3f-bad8-4894-86d7-3f760dbb506d BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G10 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-rxbkl GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf xtiflow_release-7_5_pull_cdc_integration_storage_test_285-rxbkl GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:11:54 CST 2024] <<<<<< run test case default_value success! >>>>>> [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G15 Run cases: new_ci_collation batch_add_table multi_rocks PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=2d3ecd28-7b32-4c57-8eb7-e492414f462c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G15 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-9jtg1 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8 xtiflow_release-7_5_pull_cdc_integration_storage_test_285-9jtg1 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:11:54 CST 2024] <<<<<< run test case new_ci_collation success! >>>>>> [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G12 Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=8cde64da-b9be-45ff-8e3c-9a4c890a5d53 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G12 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-gwh03 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4 xtiflow_release-7_5_pull_cdc_integration_storage_test_285-gwh03 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:11:54 CST 2024] <<<<<< run test case many_pk_or_uk success! >>>>>> [Pipeline] sh table row_format.finish_mark exists + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G04 Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=57a758f7-cb7d-4ebe-b99c-1c6e95cc838f BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G04 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-61tk3 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-61tk3 xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff successfully wait process cdc.test exit for 1-th time... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G16 wait process cdc.test exit for 2-th time... Run cases: owner_resign processor_etcd_worker_delay sink_hang PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=5b65b3d7-1a30-43f3-a39d-b4c2fd279121 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G16 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-p1kck GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms xtiflow_release-7_5_pull_cdc_integration_storage_test_285-p1kck GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:11:56 CST 2024] <<<<<< run test case row_format success! >>>>>> + check_file_exists false + local all_should_exist=false + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type/2024-04-25/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns_dummy/2024-04-25/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/binary_columns/2024-04-25/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_data_type/2022-01-01/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/multi_charset/2022-01-02/10.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/1.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/2.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/3.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/4.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/5.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/6.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/7.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/8.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/9.data ']' + for f in '${EXIST_FILES[@]}' + '[' '!' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test_default/test/binary_columns/2022-01-03/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/binary_columns/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_data_type_dummy/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test/multi_charset_dummy/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_data_type/2024-04-24/10.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/1.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/2.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/3.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/4.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/5.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/6.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/7.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/8.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/9.data ']' + for f in '${CLEANED_FILES[@]}' + '[' false == true ']' + '[' -f /tmp/tidb_cdc_test/storage_cleanup/storage_test/test2/multi_charset/2024-04-24/10.data ']' + run_sql_file /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/data/schema.sql 127.0.0.1 4000 The 1 times to try to start tidb cluster... [Pipeline] withEnv [Pipeline] { find: '/tmp/tidb_cdc_test/*/*': No such file or directory table sink_retry.finish_mark_2 exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: storage... <<================= + run_sql_file /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/data/data.sql 127.0.0.1 4000 find: '/tmp/tidb_cdc_test/*/*': No such file or directory + run_storage_consumer /tmp/tidb_cdc_test/storage_cleanup 'file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml '' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff successfully The 1 times to try to start tidb cluster... + workdir=/tmp/tidb_cdc_test/storage_cleanup wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + sink_uri='file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:11:57 CST 2024] <<<<<< run test case sink_retry success! >>>>>> + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Fri Apr 26 19:11:57 CST 2024] <<<<<< START storage consumer in storage_cleanup case >>>>>>' [Fri Apr 26 19:11:57 CST 2024] <<<<<< START storage consumer in storage_cleanup case >>>>>> + cd /tmp/tidb_cdc_test/storage_cleanup + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/storage_cleanup + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/storage_cleanup/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/storage_cleanup/storage_test?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/changefeed.toml + sleep 8 [Pipeline] { [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@14613963; decorates RemoteLauncher[hudson.remoting.Channel@67edf501:JNLP4-connect connection from 10.233.123.208/10.233.123.208:58404] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@54308251; decorates RemoteLauncher[hudson.remoting.Channel@2f4b0580:JNLP4-connect connection from 10.233.123.17/10.233.123.17:50860] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@544d9d55; decorates RemoteLauncher[hudson.remoting.Channel@5d3510d2:JNLP4-connect connection from 10.233.100.51/10.233.100.51:41864] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1fb0ed56; decorates RemoteLauncher[hudson.remoting.Channel@4ee153d1:JNLP4-connect connection from 10.233.68.164/10.233.68.164:41520] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@abe515e; decorates RemoteLauncher[hudson.remoting.Channel@356890b4:JNLP4-connect connection from 10.233.126.253/10.233.126.253:46636] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Verifying downstream PD is started... No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@50d41579; decorates RemoteLauncher[hudson.remoting.Channel@2de2581a:JNLP4-connect connection from 10.233.90.186/10.233.90.186:35950] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1a992c58; decorates RemoteLauncher[hudson.remoting.Channel@205cd41f:JNLP4-connect connection from 10.233.90.178/10.233.90.178:53754] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Cloning the remote Git repository Using shallow clone with depth 1 > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Verifying downstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Commit message: "feat(tidb): test flashbacktest package (#2942)" Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Verifying downstream PD is started... > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Verifying downstream PD is started... Verifying downstream PD is started... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:04 CST 2024] <<<<<< run test case http_api_tls success! >>>>>> Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + check_sync_diff /tmp/tidb_cdc_test/storage_cleanup /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_cleanup/conf/diff_config.toml 100 check diff failed 1-th time, retry later Commit message: "feat(tidb): test flashbacktest package (#2942)" find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: storage... <<================= > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... Commit message: "feat(tidb): test flashbacktest package (#2942)" check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Commit message: "feat(tidb): test flashbacktest package (#2942)" =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:10 CST 2024] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> check diff successfully + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Fri Apr 26 19:12:10 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>>' [Fri Apr 26 19:12:10 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c2a380002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:1303, start at 2024-04-26 19:12:10.895518906 +0800 CST m=+5.320454568 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:10.904 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:10.894 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:10.894 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c35b00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8, pid:1297, start at 2024-04-26 19:12:11.673984895 +0800 CST m=+5.355500290 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:11.682 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:11.678 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:11.678 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:12 CST 2024] <<<<<< run test case canal_json_basic success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c41ac0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:1167, start at 2024-04-26 19:12:12.424538562 +0800 CST m=+5.359479948 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:12.434 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:12.444 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:12.444 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c2a380002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:1303, start at 2024-04-26 19:12:10.895518906 +0800 CST m=+5.320454568 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:10.904 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:10.894 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:10.894 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c2af80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:1385, start at 2024-04-26 19:12:10.979930786 +0800 CST m=+5.347300604 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:10.988 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:10.992 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:10.992 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c4fb40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:1162, start at 2024-04-26 19:12:13.32413383 +0800 CST m=+5.496868783 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:13.332 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:13.293 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:13.293 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c35b00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8, pid:1297, start at 2024-04-26 19:12:11.673984895 +0800 CST m=+5.355500290 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:11.682 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:11.678 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:11.678 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c380c0011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8, pid:1377, start at 2024-04-26 19:12:11.79994384 +0800 CST m=+5.409724676 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:11.808 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:11.779 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:11.779 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c6628000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4, pid:1291, start at 2024-04-26 19:12:14.757568995 +0800 CST m=+5.469843757 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:14.766 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:14.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:14.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c4fb40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:1162, start at 2024-04-26 19:12:13.32413383 +0800 CST m=+5.496868783 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:13.332 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:13.293 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:13.293 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c4fb00014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:1233, start at 2024-04-26 19:12:13.316897874 +0800 CST m=+5.423558988 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:13.325 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:13.292 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:13.292 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2628.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c41ac0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:1167, start at 2024-04-26 19:12:12.424538562 +0800 CST m=+5.359479948 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:12.434 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:12.444 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:12.444 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c45780007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:1247, start at 2024-04-26 19:12:12.648768376 +0800 CST m=+5.494759643 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:12.657 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:12.638 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:12.638 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449348877993639937 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348877993639937 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:12:17 CST 2024] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.26662668.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 26 19:12:17 CST 2024] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.26452647.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c6628000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4, pid:1291, start at 2024-04-26 19:12:14.757568995 +0800 CST m=+5.469843757 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:14.766 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:14.730 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:14.730 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c67d40015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4, pid:1359, start at 2024-04-26 19:12:14.872722627 +0800 CST m=+5.511640593 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:14.879 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:14.837 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:14.837 +0800 All versions after safe point can be accessed. (DO NOT EDIT) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:17 CST 2024] <<<<<< run test case multi_topics success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c67400011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:1296, start at 2024-04-26 19:12:14.820994193 +0800 CST m=+5.476711910 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:14.828 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:14.800 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:14.800 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c67400011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:1296, start at 2024-04-26 19:12:14.820994193 +0800 CST m=+5.476711910 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:14.828 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:14.800 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:14.800 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c68cc0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:1362, start at 2024-04-26 19:12:14.900325569 +0800 CST m=+5.469547792 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:14.907 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:14.899 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:14.899 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 26 19:12:18 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.25382540.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2565.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449348878662369281 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348878662369281 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:12:20 CST 2024] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.26002602.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2607.out cli tso query --pd=http://127.0.0.1:2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.2633.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:20 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5333b990-4766-495f-8bc3-928b4bf4a8a7 {"id":"5333b990-4766-495f-8bc3-928b4bf4a8a7","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae1b709 5333b990-4766-495f-8bc3-928b4bf4a8a7 /tidb/cdc/default/default/upstream/7362131950411291673 {"id":7362131950411291673,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5333b990-4766-495f-8bc3-928b4bf4a8a7 {"id":"5333b990-4766-495f-8bc3-928b4bf4a8a7","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae1b709 5333b990-4766-495f-8bc3-928b4bf4a8a7 /tidb/cdc/default/default/upstream/7362131950411291673 {"id":7362131950411291673,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5333b990-4766-495f-8bc3-928b4bf4a8a7 {"id":"5333b990-4766-495f-8bc3-928b4bf4a8a7","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae1b709 5333b990-4766-495f-8bc3-928b4bf4a8a7 /tidb/cdc/default/default/upstream/7362131950411291673 {"id":7362131950411291673,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.2716.out cli changefeed create --start-ts=449348877993639937 '--sink-uri=file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-20887?protocol=canal-json&enable-tidb-extension=true' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:20 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/52e68304-8223-42a8-a86b-f4313bf59d26 {"id":"52e68304-8223-42a8-a86b-f4313bf59d26","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae2a02c 52e68304-8223-42a8-a86b-f4313bf59d26 /tidb/cdc/default/default/upstream/7362131954698820842 {"id":7362131954698820842,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/52e68304-8223-42a8-a86b-f4313bf59d26 {"id":"52e68304-8223-42a8-a86b-f4313bf59d26","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae2a02c 52e68304-8223-42a8-a86b-f4313bf59d26 /tidb/cdc/default/default/upstream/7362131954698820842 {"id":7362131954698820842,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/52e68304-8223-42a8-a86b-f4313bf59d26 {"id":"52e68304-8223-42a8-a86b-f4313bf59d26","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae2a02c 52e68304-8223-42a8-a86b-f4313bf59d26 /tidb/cdc/default/default/upstream/7362131954698820842 {"id":7362131954698820842,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.2700.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-3320?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 94170755-0018-4e00-9be1-8d486111f53e Info: {"upstream_id":7362131954698820842,"namespace":"default","id":"94170755-0018-4e00-9be1-8d486111f53e","sink_uri":"file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-3320?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:12:21.218285947+08:00","start_ts":449348879273951237,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348879273951237,"checkpoint_ts":449348879273951237,"checkpoint_time":"2024-04-26 19:12:21.078"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Create changefeed successfully! ID: d888cf0a-d4cb-4ae7-9f6d-8fd75c51b483 Info: {"upstream_id":7362131950411291673,"namespace":"default","id":"d888cf0a-d4cb-4ae7-9f6d-8fd75c51b483","sink_uri":"file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-20887?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:12:21.332646058+08:00","start_ts":449348877993639937,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348877993639937,"checkpoint_ts":449348877993639937,"checkpoint_time":"2024-04-26 19:12:16.194"} PASS ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 2.5% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:21 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/607607e1-dd6f-4982-bbb6-63ffee1385bf {"id":"607607e1-dd6f-4982-bbb6-63ffee1385bf","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7ef4 607607e1-dd6f-4982-bbb6-63ffee1385bf /tidb/cdc/default/default/upstream/7362131960816846140 {"id":7362131960816846140,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/607607e1-dd6f-4982-bbb6-63ffee1385bf {"id":"607607e1-dd6f-4982-bbb6-63ffee1385bf","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7ef4 607607e1-dd6f-4982-bbb6-63ffee1385bf /tidb/cdc/default/default/upstream/7362131960816846140 {"id":7362131960816846140,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/607607e1-dd6f-4982-bbb6-63ffee1385bf {"id":"607607e1-dd6f-4982-bbb6-63ffee1385bf","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7ef4 607607e1-dd6f-4982-bbb6-63ffee1385bf /tidb/cdc/default/default/upstream/7362131960816846140 {"id":7362131960816846140,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.2626.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: 0f33908e-c934-4c94-886c-14ccc2217846 Info: {"upstream_id":7362131960816846140,"namespace":"default","id":"0f33908e-c934-4c94-886c-14ccc2217846","sink_uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:12:22.028241366+08:00","start_ts":449348879500705794,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348879500705794,"checkpoint_ts":449348879500705794,"checkpoint_time":"2024-04-26 19:12:21.943"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + tso='449348879201075206 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348879201075206 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:12:22 CST 2024] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.26502652.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + workdir=/tmp/tidb_cdc_test/batch_add_table + sink_uri='file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-3320?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/batch_add_table ++ date + echo '[Fri Apr 26 19:12:22 CST 2024] <<<<<< START storage consumer in batch_add_table case >>>>>>' [Fri Apr 26 19:12:22 CST 2024] <<<<<< START storage consumer in batch_add_table case >>>>>> + cd /tmp/tidb_cdc_test/batch_add_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/batch_add_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/batch_add_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/batch_add_table/storage_test/ticdc-batch-add-table-test-3320?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + tso='449348879287582721 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348879287582721 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:12:22 CST 2024] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.26672669.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_add_table.finish_mark not exists for 1-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/simple + sink_uri='file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-20887?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/simple ++ date + echo '[Fri Apr 26 19:12:22 CST 2024] <<<<<< START storage consumer in simple case >>>>>>' [Fri Apr 26 19:12:22 CST 2024] <<<<<< START storage consumer in simple case >>>>>> + cd /tmp/tidb_cdc_test/simple + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/simple + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/simple/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/simple/storage_test/ticdc-simple-test-20887?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' succeed to verify meta placement rules ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:22 CST 2024] <<<<<< run test case avro_basic success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:23 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3935d02f-7358-42d0-82b7-900d7ca06570 {"id":"3935d02f-7358-42d0-82b7-900d7ca06570","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae59ef5 3935d02f-7358-42d0-82b7-900d7ca06570 /tidb/cdc/default/default/upstream/7362131956816712018 {"id":7362131956816712018,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3935d02f-7358-42d0-82b7-900d7ca06570 {"id":"3935d02f-7358-42d0-82b7-900d7ca06570","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae59ef5 3935d02f-7358-42d0-82b7-900d7ca06570 /tidb/cdc/default/default/upstream/7362131956816712018 {"id":7362131956816712018,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3935d02f-7358-42d0-82b7-900d7ca06570 {"id":"3935d02f-7358-42d0-82b7-900d7ca06570","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ae59ef5 3935d02f-7358-42d0-82b7-900d7ca06570 /tidb/cdc/default/default/upstream/7362131956816712018 {"id":7362131956816712018,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2664.out cli changefeed create --start-ts=449348878662369281 '--sink-uri=file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-5962?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: e6c2a23b-6e41-437b-bc46-7e742b75bdb4 Info: {"upstream_id":7362131956816712018,"namespace":"default","id":"e6c2a23b-6e41-437b-bc46-7e742b75bdb4","sink_uri":"file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-5962?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:12:23.622230963+08:00","start_ts":449348878662369281,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348878662369281,"checkpoint_ts":449348878662369281,"checkpoint_time":"2024-04-26 19:12:18.745"} PASS + set +x + workdir=/tmp/tidb_cdc_test/owner_resign + sink_uri='file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/owner_resign ++ date + echo '[Fri Apr 26 19:12:23 CST 2024] <<<<<< START storage consumer in owner_resign case >>>>>>' [Fri Apr 26 19:12:23 CST 2024] <<<<<< START storage consumer in owner_resign case >>>>>> + cd /tmp/tidb_cdc_test/owner_resign + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/owner_resign + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/owner_resign/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686cc6b80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:3748, start at 2024-04-26 19:12:20.940959997 +0800 CST m=+5.285127002 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:20.948 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:20.910 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:20.910 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686cc6b80012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:3748, start at 2024-04-26 19:12:20.940959997 +0800 CST m=+5.285127002 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:20.948 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:20.910 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:20.910 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686cc9080014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:3818, start at 2024-04-26 19:12:21.098136335 +0800 CST m=+5.388117407 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:21.105 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:21.108 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:21.108 +0800 All versions after safe point can be accessed. (DO NOT EDIT) coverage: 2.5% of statements in github.com/pingcap/tiflow/... 1 run task successfully table owner_resign.t1 not exists for 1-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table batch_add_table.finish_mark not exists for 2-th check, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 2-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/foreign_key + sink_uri='file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-5962?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/foreign_key ++ date + echo '[Fri Apr 26 19:12:25 CST 2024] <<<<<< START storage consumer in foreign_key case >>>>>>' [Fri Apr 26 19:12:25 CST 2024] <<<<<< START storage consumer in foreign_key case >>>>>> + cd /tmp/tidb_cdc_test/foreign_key + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/foreign_key + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/foreign_key/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/foreign_key/storage_test/ticdc-foreign-key-test-5962?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:25 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2cbeab83-e43f-4a86-a805-ec2badd2721c {"id":"2cbeab83-e43f-4a86-a805-ec2badd2721c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aec79f7 2cbeab83-e43f-4a86-a805-ec2badd2721c /tidb/cdc/default/default/upstream/7362131966098464248 {"id":7362131966098464248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2cbeab83-e43f-4a86-a805-ec2badd2721c {"id":"2cbeab83-e43f-4a86-a805-ec2badd2721c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aec79f7 2cbeab83-e43f-4a86-a805-ec2badd2721c /tidb/cdc/default/default/upstream/7362131966098464248 {"id":7362131966098464248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2cbeab83-e43f-4a86-a805-ec2badd2721c {"id":"2cbeab83-e43f-4a86-a805-ec2badd2721c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aec79f7 2cbeab83-e43f-4a86-a805-ec2badd2721c /tidb/cdc/default/default/upstream/7362131966098464248 {"id":7362131966098464248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2702.out cli changefeed create --start-ts=449348879201075206 '--sink-uri=file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-5537?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: f665948a-e0d6-42af-8879-71223683e3e3 Info: {"upstream_id":7362131966098464248,"namespace":"default","id":"f665948a-e0d6-42af-8879-71223683e3e3","sink_uri":"file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-5537?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:12:25.650667464+08:00","start_ts":449348879201075206,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348879201075206,"checkpoint_ts":449348879201075206,"checkpoint_time":"2024-04-26 19:12:20.800"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ce0400008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:3780, start at 2024-04-26 19:12:22.551444292 +0800 CST m=+5.247151671 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:22.558 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:22.544 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:22.544 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ce0400008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:3780, start at 2024-04-26 19:12:22.551444292 +0800 CST m=+5.247151671 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:22.558 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:22.544 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:22.544 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ce1c4000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:3844, start at 2024-04-26 19:12:22.652118716 +0800 CST m=+5.281029047 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:22.660 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:22.641 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:22.641 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:25 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8aa192f-f28a-434a-a078-df8ee6dd83af {"id":"d8aa192f-f28a-434a-a078-df8ee6dd83af","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1af01d0d d8aa192f-f28a-434a-a078-df8ee6dd83af /tidb/cdc/default/default/upstream/7362131965193431600 {"id":7362131965193431600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8aa192f-f28a-434a-a078-df8ee6dd83af {"id":"d8aa192f-f28a-434a-a078-df8ee6dd83af","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1af01d0d d8aa192f-f28a-434a-a078-df8ee6dd83af /tidb/cdc/default/default/upstream/7362131965193431600 {"id":7362131965193431600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d8aa192f-f28a-434a-a078-df8ee6dd83af {"id":"d8aa192f-f28a-434a-a078-df8ee6dd83af","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1af01d0d d8aa192f-f28a-434a-a078-df8ee6dd83af /tidb/cdc/default/default/upstream/7362131965193431600 {"id":7362131965193431600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table owner_resign.t1 not exists for 2-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5115.out cli tso query --pd=http://127.0.0.1:2379 + workdir=/tmp/tidb_cdc_test/capture_session_done_during_task + sink_uri='file:///tmp/tidb_cdc_test/capture_session_done_during_task/storage_test/ticdc-capture-session-done-during-task-19433?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/capture_session_done_during_task ++ date + echo '[Fri Apr 26 19:12:26 CST 2024] <<<<<< START storage consumer in capture_session_done_during_task case >>>>>>' [Fri Apr 26 19:12:26 CST 2024] <<<<<< START storage consumer in capture_session_done_during_task case >>>>>> + cd /tmp/tidb_cdc_test/capture_session_done_during_task + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/capture_session_done_during_task + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/capture_session_done_during_task/storage_test/ticdc-capture-session-done-during-task-19433?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' lease 22318f1a1af01d0d revoked table foreign_key.finish_mark not exists for 1-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/resolve_lock + sink_uri='file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-5537?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/resolve_lock ++ date + echo '[Fri Apr 26 19:12:27 CST 2024] <<<<<< START storage consumer in resolve_lock case >>>>>>' [Fri Apr 26 19:12:27 CST 2024] <<<<<< START storage consumer in resolve_lock case >>>>>> + cd /tmp/tidb_cdc_test/resolve_lock + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/resolve_lock + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/resolve_lock/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/resolve_lock/storage_test/ticdc-resolve-lock-test-5537?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 table batch_add_table.finish_mark not exists for 3-th check, retry later go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/btree v1.1.2 go: downloading google.golang.org/protobuf v1.33.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 3-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5135.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449348880645226497 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348880645226497 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading golang.org/x/sys v0.19.0 table capture_session_done_during_task.t exists go: downloading golang.org/x/net v0.24.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 check diff failed 1-th time, retry later table owner_resign.t1 not exists for 3-th check, retry later go: downloading golang.org/x/text v0.14.0 ***************** properties ***************** "mysql.db"="changefeed_error" "workload"="core" "mysql.user"="root" "requestdistribution"="uniform" "scanproportion"="0" "recordcount"="20" "mysql.host"="127.0.0.1" "operationcount"="0" "readallfields"="true" "readproportion"="0" "threadcount"="4" "mysql.port"="4000" "insertproportion"="0" "dotransactions"="false" "updateproportion"="0" ********************************************** Run finished, takes 28.542237ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3043.9, Avg(us): 5533, Min(us): 1053, Max(us): 21946, 95th(us): 22000, 99th(us): 22000 [Fri Apr 26 19:12:28 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.51745176.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table batch_add_table.finish_mark not exists for 4-th check, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 4-th time, retry later table foreign_key.finish_mark not exists for 2-th check, retry later + set +x + tso='449348881073831937 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348881073831937 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:12:29 CST 2024] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.51635165.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table owner_resign.t1 not exists for 4-th check, retry later go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/klauspost/compress v1.17.7 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading google.golang.org/api v0.162.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading cloud.google.com/go v0.112.0 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/Masterminds/semver v1.5.0 go: downloading k8s.io/api v0.28.6 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/rivo/uniseg v0.4.6 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading cloud.google.com/go/compute v1.24.0 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/tklauser/numcpus v0.6.1 check diff failed 2-th time, retry later go: downloading cloud.google.com/go/iam v1.1.6 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 table batch_add_table.finish_mark not exists for 5-th check, retry later ERROR 1146 (42S02) at line 1: Table 'test.simple1' doesn't exist check data failed 5-th time, retry later table foreign_key.finish_mark not exists for 3-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:30 CST 2024] <<<<<< run test case canal_json_handle_key_only success! >>>>>> go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading k8s.io/apimachinery v0.28.6 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b table owner_resign.t1 not exists for 5-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:31 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 {"id":"fd6d0135-09ab-4880-a221-49090107dd42","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d220a fd6d0135-09ab-4880-a221-49090107dd42 /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 {"id":"fd6d0135-09ab-4880-a221-49090107dd42","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d220a fd6d0135-09ab-4880-a221-49090107dd42 /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 {"id":"fd6d0135-09ab-4880-a221-49090107dd42","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d220a fd6d0135-09ab-4880-a221-49090107dd42 /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5222.out cli changefeed create --start-ts=449348880645226497 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7362132000486624707,"namespace":"default","id":"changefeed-error","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:12:32.353528801+08:00","start_ts":449348880645226497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348880645226497,"checkpoint_ts":449348880645226497,"checkpoint_time":"2024-04-26 19:12:26.309"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/csv_storage_basic Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:32 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34268fbb-6977-4016-b018-f0703dc658e5 {"id":"34268fbb-6977-4016-b018-f0703dc658e5","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0bb3f6 34268fbb-6977-4016-b018-f0703dc658e5 /tidb/cdc/default/default/upstream/7362132008438403920 {"id":7362132008438403920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34268fbb-6977-4016-b018-f0703dc658e5 {"id":"34268fbb-6977-4016-b018-f0703dc658e5","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0bb3f6 34268fbb-6977-4016-b018-f0703dc658e5 /tidb/cdc/default/default/upstream/7362132008438403920 {"id":7362132008438403920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/34268fbb-6977-4016-b018-f0703dc658e5 {"id":"34268fbb-6977-4016-b018-f0703dc658e5","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0bb3f6 34268fbb-6977-4016-b018-f0703dc658e5 /tidb/cdc/default/default/upstream/7362132008438403920 {"id":7362132008438403920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.5216.out cli changefeed create --start-ts=449348881073831937 '--sink-uri=file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-7392?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: dad90375-18dc-449b-afb2-f064e330df2e Info: {"upstream_id":7362132008438403920,"namespace":"default","id":"dad90375-18dc-449b-afb2-f064e330df2e","sink_uri":"file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-7392?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:12:32.809877211+08:00","start_ts":449348881073831937,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348881073831937,"checkpoint_ts":449348881073831937,"checkpoint_time":"2024-04-26 19:12:27.944"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table batch_add_table.finish_mark not exists for 6-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 6-th time, retry later table foreign_key.finish_mark not exists for 4-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:33 CST 2024] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> + set +x + workdir=/tmp/tidb_cdc_test/changefeed_error + sink_uri='file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_error ++ date + echo '[Fri Apr 26 19:12:33 CST 2024] <<<<<< START storage consumer in changefeed_error case >>>>>>' [Fri Apr 26 19:12:33 CST 2024] <<<<<< START storage consumer in changefeed_error case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_error + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_error + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_error/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s table owner_resign.t1 exists check diff failed 1-th time, retry later + info='{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449348880645226497, "checkpoint_time": "2024-04-26 19:12:26.309", "error": { "time": "2024-04-26T19:12:32.42243633+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348880645226197 is earlier than or equal to GC safepoint at 449348880645226497" } }' + echo '{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449348880645226497, "checkpoint_time": "2024-04-26 19:12:26.309", "error": { "time": "2024-04-26T19:12:32.42243633+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348880645226197 is earlier than or equal to GC safepoint at 449348880645226497" } }' { "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449348880645226497, "checkpoint_time": "2024-04-26 19:12:26.309", "error": { "time": "2024-04-26T19:12:32.42243633+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348880645226197 is earlier than or equal to GC safepoint at 449348880645226497" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 449348880645226497, '"checkpoint_time":' '"2024-04-26' '19:12:26.309",' '"error":' '{' '"time":' '"2024-04-26T19:12:32.42243633+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 449348880645226197 is earlier than or equal to GC safepoint at '449348880645226497"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 449348880645226497, '"checkpoint_time":' '"2024-04-26' '19:12:26.309",' '"error":' '{' '"time":' '"2024-04-26T19:12:32.42243633+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 449348880645226197 is earlier than or equal to GC safepoint at '449348880645226497"' '}' '}' + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348880645226197 is earlier than or equal to GC safepoint at 449348880645226497' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348880645226197 is earlier than or equal to GC safepoint at 449348880645226497 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5326.out cli changefeed resume -c changefeed-error + set +x + workdir=/tmp/tidb_cdc_test/drop_many_tables + sink_uri='file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-7392?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/drop_many_tables ++ date + echo '[Fri Apr 26 19:12:34 CST 2024] <<<<<< START storage consumer in drop_many_tables case >>>>>>' [Fri Apr 26 19:12:34 CST 2024] <<<<<< START storage consumer in drop_many_tables case >>>>>> + cd /tmp/tidb_cdc_test/drop_many_tables + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/drop_many_tables + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/drop_many_tables/storage_test/ticdc-drop-tables-test-7392?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 4-th time, retry later PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... table batch_add_table.finish_mark exists check diff successfully go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 check data successfully table foreign_key.finish_mark not exists for 5-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 1-th time... + set +x table changefeed_error.usertable not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... check diff failed 2-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:12:36 CST 2024] <<<<<< run test case batch_add_table success! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:36 CST 2024] <<<<<< run test case canal_json_claim_check success! >>>>>> Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:12:36 CST 2024] <<<<<< run test case simple success! >>>>>> check diff failed 5-th time, retry later table foreign_key.finish_mark not exists for 6-th check, retry later Starting Upstream TiDB... check diff failed 3-th time, retry later table drop_tables.c not exists for 1-th check, retry later Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_error.usertable not exists for 2-th check, retry later check diff successfully check diff failed 1-th time, retry later table foreign_key.finish_mark not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_error.usertable not exists for 3-th check, retry later check diff failed 4-th time, retry later table drop_tables.c not exists for 2-th check, retry later check diff failed 2-th time, retry later table foreign_key.finish_mark not exists for 8-th check, retry later table changefeed_error.usertable not exists for 4-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 36.92 secs (91142555 bytes/sec) [Pipeline] { check diff failed 5-th time, retry later table drop_tables.c not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later table foreign_key.finish_mark not exists for 9-th check, retry later table changefeed_error.usertable exists check diff successfully ***************** properties ***************** "threadcount"="4" "requestdistribution"="uniform" "workload"="core" "updateproportion"="0" "operationcount"="0" "recordcount"="20" "mysql.host"="127.0.0.1" "readproportion"="0" "readallfields"="true" "mysql.port"="4000" "scanproportion"="0" "dotransactions"="false" "insertproportion"="0" "mysql.user"="root" "mysql.db"="changefeed_error" ********************************************** Run finished, takes 4.833752ms INSERT - Takes(s): 0.0, Count: 20, OPS: 5878.4, Avg(us): 809, Min(us): 466, Max(us): 1847, 95th(us): 2000, 99th(us): 2000 check diff successfully {"id":"fd6d0135-09ab-4880-a221-49090107dd42","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 + [[ /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later check diff successfully [Fri Apr 26 19:12:44 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.29962998.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table drop_tables.c not exists for 4-th check, retry later check diff failed 4-th time, retry later table foreign_key.finish_mark not exists for 10-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686e2a980012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:4974, start at 2024-04-26 19:12:43.725099457 +0800 CST m=+5.484099637 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:43.732 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:43.735 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:43.735 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686e2a980012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:4974, start at 2024-04-26 19:12:43.725099457 +0800 CST m=+5.484099637 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:43.732 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:43.735 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:43.735 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686e2c080014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:5050, start at 2024-04-26 19:12:43.816116339 +0800 CST m=+5.495175010 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:43.824 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:43.828 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:43.828 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_basic/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 + [[ /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later table drop_tables.c not exists for 5-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:12:45 CST 2024] <<<<<< run test case open_protocol_claim_check success! >>>>>> [Fri Apr 26 19:12:47 CST 2024] <<<<<< START cdc server in csv_storage_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_basic.62896291.out server --log-file /tmp/tidb_cdc_test/csv_storage_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_basic/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully wait process cdc.test exit for 1-th time... table foreign_key.finish_mark not exists for 11-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:47 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0f33908e-c934-4c94-886c-14ccc2217846 {UpstreamID:7362131960816846140 Namespace:default ID:0f33908e-c934-4c94-886c-14ccc2217846 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:12:22.028241366 +0800 CST StartTs:449348879500705794 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c11290 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348879513812997} {CheckpointTs:449348886054305802 MinTableBarrierTs:449348886054305802 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/445c61d0-f2d7-4671-b814-a80dc7325cef {"id":"445c61d0-f2d7-4671-b814-a80dc7325cef","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/607607e1-dd6f-4982-bbb6-63ffee1385bf {"id":"607607e1-dd6f-4982-bbb6-63ffee1385bf","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7ef4 607607e1-dd6f-4982-bbb6-63ffee1385bf /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7fba 445c61d0-f2d7-4671-b814-a80dc7325cef /tidb/cdc/default/default/changefeed/info/0f33908e-c934-4c94-886c-14ccc2217846 {"upstream-id":7362131960816846140,"namespace":"default","changefeed-id":"0f33908e-c934-4c94-886c-14ccc2217846","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:12:22.028241366+08:00","start-ts":449348879500705794,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348879513812997} /tidb/cdc/default/default/changefeed/status/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":449348886054305802,"min-table-barrier-ts":449348886054305802,"admin-job-type":0} /tidb/cdc/default/default/task/position/445c61d0-f2d7-4671-b814-a80dc7325cef/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/607607e1-dd6f-4982-bbb6-63ffee1385bf/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362131960816846140 {"id":7362131960816846140,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0f33908e-c934-4c94-886c-14ccc2217846 {UpstreamID:7362131960816846140 Namespace:default ID:0f33908e-c934-4c94-886c-14ccc2217846 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:12:22.028241366 +0800 CST StartTs:449348879500705794 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c11290 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348879513812997} {CheckpointTs:449348886054305802 MinTableBarrierTs:449348886054305802 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/445c61d0-f2d7-4671-b814-a80dc7325cef {"id":"445c61d0-f2d7-4671-b814-a80dc7325cef","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/607607e1-dd6f-4982-bbb6-63ffee1385bf {"id":"607607e1-dd6f-4982-bbb6-63ffee1385bf","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7ef4 607607e1-dd6f-4982-bbb6-63ffee1385bf /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7fba 445c61d0-f2d7-4671-b814-a80dc7325cef /tidb/cdc/default/default/changefeed/info/0f33908e-c934-4c94-886c-14ccc2217846 {"upstream-id":7362131960816846140,"namespace":"default","changefeed-id":"0f33908e-c934-4c94-886c-14ccc2217846","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:12:22.028241366+08:00","start-ts":449348879500705794,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348879513812997} /tidb/cdc/default/default/changefeed/status/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":449348886054305802,"min-table-barrier-ts":449348886054305802,"admin-job-type":0} /tidb/cdc/default/default/task/position/445c61d0-f2d7-4671-b814-a80dc7325cef/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/607607e1-dd6f-4982-bbb6-63ffee1385bf/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362131960816846140 {"id":7362131960816846140,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/0f33908e-c934-4c94-886c-14ccc2217846 {UpstreamID:7362131960816846140 Namespace:default ID:0f33908e-c934-4c94-886c-14ccc2217846 SinkURI:file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:12:22.028241366 +0800 CST StartTs:449348879500705794 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c11290 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348879513812997} {CheckpointTs:449348886054305802 MinTableBarrierTs:449348886054305802 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/445c61d0-f2d7-4671-b814-a80dc7325cef {"id":"445c61d0-f2d7-4671-b814-a80dc7325cef","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/607607e1-dd6f-4982-bbb6-63ffee1385bf {"id":"607607e1-dd6f-4982-bbb6-63ffee1385bf","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7ef4 607607e1-dd6f-4982-bbb6-63ffee1385bf /tidb/cdc/default/__cdc_meta__/owner/22318f1a1aea7fba 445c61d0-f2d7-4671-b814-a80dc7325cef /tidb/cdc/default/default/changefeed/info/0f33908e-c934-4c94-886c-14ccc2217846 {"upstream-id":7362131960816846140,"namespace":"default","changefeed-id":"0f33908e-c934-4c94-886c-14ccc2217846","sink-uri":"file:///tmp/tidb_cdc_test/owner_resign/storage_test/ticdc-owner-resign-test-19333?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:12:22.028241366+08:00","start-ts":449348879500705794,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348879513812997} /tidb/cdc/default/default/changefeed/status/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":449348886054305802,"min-table-barrier-ts":449348886054305802,"admin-job-type":0} /tidb/cdc/default/default/task/position/445c61d0-f2d7-4671-b814-a80dc7325cef/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/607607e1-dd6f-4982-bbb6-63ffee1385bf/0f33908e-c934-4c94-886c-14ccc2217846 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362131960816846140 {"id":7362131960816846140,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:+ grep -q 'etcd info' 2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 wait process cdc.test exit for 2-th time... table drop_tables.c not exists for 6-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 3-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:12:49 CST 2024] <<<<<< run test case capture_session_done_during_task success! >>>>>> table foreign_key.finish_mark not exists for 12-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 + [[ /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/fd6d0135-09ab-4880-a221-49090107dd42 + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:50 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/42c3a841-0563-4829-b201-5255e4e2468f {"id":"42c3a841-0563-4829-b201-5255e4e2468f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b5f75f2 42c3a841-0563-4829-b201-5255e4e2468f /tidb/cdc/default/default/upstream/7362132101072161859 {"id":7362132101072161859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/42c3a841-0563-4829-b201-5255e4e2468f {"id":"42c3a841-0563-4829-b201-5255e4e2468f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b5f75f2 42c3a841-0563-4829-b201-5255e4e2468f /tidb/cdc/default/default/upstream/7362132101072161859 {"id":7362132101072161859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/42c3a841-0563-4829-b201-5255e4e2468f {"id":"42c3a841-0563-4829-b201-5255e4e2468f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b5f75f2 42c3a841-0563-4829-b201-5255e4e2468f /tidb/cdc/default/default/upstream/7362132101072161859 {"id":7362132101072161859,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_basic.cli.6332.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml start tidb cluster in /tmp/tidb_cdc_test/multi_rocks Starting Upstream PD... Create changefeed successfully! ID: 84a0aaab-fef9-4bea-8690-c8b19b9e4cc9 Info: {"upstream_id":7362132101072161859,"namespace":"default","id":"84a0aaab-fef9-4bea-8690-c8b19b9e4cc9","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s","create_time":"2024-04-26T19:12:50.558716066+08:00","start_ts":449348886983081989,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348886983081989,"checkpoint_ts":449348886983081989,"checkpoint_time":"2024-04-26 19:12:50.486"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table drop_tables.c not exists for 7-th check, retry later Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... [Pipeline] sh Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { table foreign_key.finish_mark not exists for 13-th check, retry later [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G07 Run cases: kv_client_stream_reconnect cdc split_region PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=027a8817-7abd-4ed1-b69e-e1b308a8086a BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G07 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-8tl6s GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-8tl6s xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/canal_json_storage_basic Starting Upstream PD... check diff failed 1-th time, retry later Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... + set +x + workdir=/tmp/tidb_cdc_test/csv_storage_basic + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_basic ++ date + echo '[Fri Apr 26 19:12:52 CST 2024] <<<<<< START storage consumer in csv_storage_basic case >>>>>>' [Fri Apr 26 19:12:52 CST 2024] <<<<<< START storage consumer in csv_storage_basic case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_basic + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_basic + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_basic/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_basic/storage_test?flush-interval=5s' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_basic/conf/changefeed.toml [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { table drop_tables.c not exists for 8-th check, retry later [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) table foreign_key.finish_mark not exists for 14-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials check diff failed 2-th time, retry later Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow [Pipeline] { [Pipeline] cache table drop_tables.c not exists for 9-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table foreign_key.finish_mark not exists for 15-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later table drop_tables.c exists check diff successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 26 19:12:56 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.55375539.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table foreign_key.finish_mark not exists for 16-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:12:58 CST 2024] <<<<<< run test case drop_many_tables success! >>>>>> check diff failed 4-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:59 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c17e08c-4218-4e9d-8cdc-7116afe0f56d {"id":"2c17e08c-4218-4e9d-8cdc-7116afe0f56d","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d22aa 2c17e08c-4218-4e9d-8cdc-7116afe0f56d /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7362132000486624707,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:12:32.353528801+08:00","start-ts":449348880645226497,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-04-26T19:12:56.81331348+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348888640880642} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449348885206532100,"min-table-barrier-ts":449348885206532101,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c17e08c-4218-4e9d-8cdc-7116afe0f56d {"id":"2c17e08c-4218-4e9d-8cdc-7116afe0f56d","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d22aa 2c17e08c-4218-4e9d-8cdc-7116afe0f56d /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7362132000486624707,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:12:32.353528801+08:00","start-ts":449348880645226497,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-04-26T19:12:56.81331348+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348888640880642} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449348885206532100,"min-table-barrier-ts":449348885206532101,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c17e08c-4218-4e9d-8cdc-7116afe0f56d {"id":"2c17e08c-4218-4e9d-8cdc-7116afe0f56d","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d22aa 2c17e08c-4218-4e9d-8cdc-7116afe0f56d /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7362132000486624707,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:12:32.353528801+08:00","start-ts":449348880645226497,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-04-26T19:12:56.81331348+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348888640880642} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449348885206532100,"min-table-barrier-ts":449348885206532101,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s table foreign_key.finish_mark not exists for 17-th check, retry later + info='{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449348885206532100, "checkpoint_time": "2024-04-26 19:12:43.709", "error": { "time": "2024-04-26T19:12:56.81331348+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449348885206532100, "checkpoint_time": "2024-04-26 19:12:43.709", "error": { "time": "2024-04-26T19:12:56.81331348+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449348885206532100, "checkpoint_time": "2024-04-26 19:12:43.709", "error": { "time": "2024-04-26T19:12:56.81331348+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 449348885206532100, '"checkpoint_time":' '"2024-04-26' '19:12:43.709",' '"error":' '{' '"time":' '"2024-04-26T19:12:56.81331348+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 449348885206532100, '"checkpoint_time":' '"2024-04-26' '19:12:43.709",' '"error":' '{' '"time":' '"2024-04-26T19:12:56.81331348+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5630.out cli changefeed remove -c changefeed-error ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later check diff failed 1-th time, retry later Changefeed remove successfully. ID: changefeed-error CheckpointTs: 449348885206532100 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f31a80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:3860, start at 2024-04-26 19:13:00.550349128 +0800 CST m=+5.384587033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:00.558 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:00.522 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:00.522 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table foreign_key.finish_mark not exists for 18-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully wait process cdc.test exit for 1-th time... check diff failed 6-th time, retry later check diff failed 2-th time, retry later wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f31a80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:3860, start at 2024-04-26 19:13:00.550349128 +0800 CST m=+5.384587033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:00.558 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:00.522 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:00.522 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f33c80015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:3941, start at 2024-04-26 19:13:00.703414277 +0800 CST m=+5.476666290 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:00.710 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:00.708 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:00.708 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 26 19:13:03 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.57115713.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' table foreign_key.finish_mark not exists for 19-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 check diff successfully [Fri Apr 26 19:13:04 CST 2024] <<<<<< run test case csv_storage_basic success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f65e80016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9, pid:2089, start at 2024-04-26 19:13:03.896652208 +0800 CST m=+5.195081057 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:03.903 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:03.866 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:03.866 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f65e80016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9, pid:2089, start at 2024-04-26 19:13:03.896652208 +0800 CST m=+5.195081057 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:03.903 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:03.866 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:03.866 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f67a40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9, pid:2167, start at 2024-04-26 19:13:04.022434977 +0800 CST m=+5.264415732 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:04.030 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:04.027 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:04.027 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_storage_basic/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table foreign_key.finish_mark not exists for 20-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.5247.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f76640017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9, pid:1166, start at 2024-04-26 19:13:04.947430638 +0800 CST m=+5.190405043 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:04.955 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:04.921 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:04.921 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f76640017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9, pid:1166, start at 2024-04-26 19:13:04.947430638 +0800 CST m=+5.190405043 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:04.955 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:04.921 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:04.921 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f77100014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9, pid:1258, start at 2024-04-26 19:13:05.011475979 +0800 CST m=+5.200879708 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:05.018 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:05.014 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:05.014 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 26 19:13:07 CST 2024] <<<<<< START cdc server in canal_json_storage_basic case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_basic.34113413.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_basic/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_basic/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:06 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf8be4b9-60cf-4687-8523-e40ddfd4f99f {"id":"bf8be4b9-60cf-4687-8523-e40ddfd4f99f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d22e0 bf8be4b9-60cf-4687-8523-e40ddfd4f99f /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf8be4b9-60cf-4687-8523-e40ddfd4f99f {"id":"bf8be4b9-60cf-4687-8523-e40ddfd4f99f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d22e0 bf8be4b9-60cf-4687-8523-e40ddfd4f99f /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bf8be4b9-60cf-4687-8523-e40ddfd4f99f {"id":"bf8be4b9-60cf-4687-8523-e40ddfd4f99f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d22e0 bf8be4b9-60cf-4687-8523-e40ddfd4f99f /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5758.out cli changefeed create --start-ts=449348880645226497 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-1 Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7362132000486624707,"namespace":"default","id":"changefeed-error-1","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:13:06.898972649+08:00","start_ts":449348880645226497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348880645226497,"checkpoint_ts":449348880645226497,"checkpoint_time":"2024-04-26 19:12:26.309"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 8.68 secs (387773440 bytes/sec) [Pipeline] { [Pipeline] cache table foreign_key.finish_mark exists + set +x + tso='449348891003322369 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348891003322369 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x try a VALID cdc server command [Fri Apr 26 19:13:07 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.52855287.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... wait process cdc.test exit for 1-th time... [Fri Apr 26 19:13:08 CST 2024] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.25592561.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed process cdc.test already exit [Fri Apr 26 19:13:09 CST 2024] <<<<<< run test case foreign_key success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 1610 0 --:--:-- --:--:-- --:--:-- 1621 + info='{"state":"warning","resolved_ts":449348881038442511,"checkpoint_ts":449348881038442511,"last_warning":{"time":"2024-04-26T19:13:08.891214666+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":449348881038442511,"checkpoint_ts":449348881038442511,"last_warning":{"time":"2024-04-26T19:13:08.891214666+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":449348881038442511,"checkpoint_ts":449348881038442511,"last_warning":{"time":"2024-04-26T19:13:08.891214666+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ echo '{"state":"warning","resolved_ts":449348881038442511,"checkpoint_ts":449348881038442511,"last_warning":{"time":"2024-04-26T19:13:08.891214666+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ jq -r .last_warning.message ++ echo '{"state":"warning","resolved_ts":449348881038442511,"checkpoint_ts":449348881038442511,"last_warning":{"time":"2024-04-26T19:13:08.891214666+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5813.out cli changefeed remove -c changefeed-error-1 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 449348881038442511 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:10 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bfe607f4-fad2-41d9-9e73-dd54f860d5e3 {"id":"bfe607f4-fad2-41d9-9e73-dd54f860d5e3","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1baa96f2 bfe607f4-fad2-41d9-9e73-dd54f860d5e3 /tidb/cdc/default/default/upstream/7362132175901921128 {"id":7362132175901921128,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bfe607f4-fad2-41d9-9e73-dd54f860d5e3 {"id":"bfe607f4-fad2-41d9-9e73-dd54f860d5e3","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1baa96f2 bfe607f4-fad2-41d9-9e73-dd54f860d5e3 /tidb/cdc/default/default/upstream/7362132175901921128 {"id":7362132175901921128,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bfe607f4-fad2-41d9-9e73-dd54f860d5e3 {"id":"bfe607f4-fad2-41d9-9e73-dd54f860d5e3","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1baa96f2 bfe607f4-fad2-41d9-9e73-dd54f860d5e3 /tidb/cdc/default/default/upstream/7362132175901921128 {"id":7362132175901921128,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_basic.cli.3458.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Create changefeed successfully! ID: a38ac9eb-ce5d-496b-9343-1fddd12b04f2 Info: {"upstream_id":7362132175901921128,"namespace":"default","id":"a38ac9eb-ce5d-496b-9343-1fddd12b04f2","sink_uri":"file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:13:10.518867364+08:00","start_ts":449348892207611907,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348892207611907,"checkpoint_ts":449348892207611907,"checkpoint_time":"2024-04-26 19:13:10.416"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:10 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bb233ddc-74e7-4df0-b1ea-b0577740b800 {"id":"bb233ddc-74e7-4df0-b1ea-b0577740b800","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba77b00 bb233ddc-74e7-4df0-b1ea-b0577740b800 /tidb/cdc/default/default/upstream/7362132174677740998 {"id":7362132174677740998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bb233ddc-74e7-4df0-b1ea-b0577740b800 {"id":"bb233ddc-74e7-4df0-b1ea-b0577740b800","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba77b00 bb233ddc-74e7-4df0-b1ea-b0577740b800 /tidb/cdc/default/default/upstream/7362132174677740998 {"id":7362132174677740998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bb233ddc-74e7-4df0-b1ea-b0577740b800 {"id":"bb233ddc-74e7-4df0-b1ea-b0577740b800","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba77b00 bb233ddc-74e7-4df0-b1ea-b0577740b800 /tidb/cdc/default/default/upstream/7362132174677740998 {"id":7362132174677740998,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:11 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5ecb4ce2-81b5-430c-8f05-bb9a073a0403 {"id":"5ecb4ce2-81b5-430c-8f05-bb9a073a0403","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bb588f0 5ecb4ce2-81b5-430c-8f05-bb9a073a0403 /tidb/cdc/default/default/upstream/7362132192239069368 {"id":7362132192239069368,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5ecb4ce2-81b5-430c-8f05-bb9a073a0403 {"id":"5ecb4ce2-81b5-430c-8f05-bb9a073a0403","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bb588f0 5ecb4ce2-81b5-430c-8f05-bb9a073a0403 /tidb/cdc/default/default/upstream/7362132192239069368 {"id":7362132192239069368,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5ecb4ce2-81b5-430c-8f05-bb9a073a0403 {"id":"5ecb4ce2-81b5-430c-8f05-bb9a073a0403","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bb588f0 5ecb4ce2-81b5-430c-8f05-bb9a073a0403 /tidb/cdc/default/default/upstream/7362132192239069368 {"id":7362132192239069368,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/kv_client_stream_reconnect + sink_uri='file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/storage_test/kv-client-stream-reconnect-31843?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/kv_client_stream_reconnect ++ date + echo '[Fri Apr 26 19:13:11 CST 2024] <<<<<< START storage consumer in kv_client_stream_reconnect case >>>>>>' [Fri Apr 26 19:13:11 CST 2024] <<<<<< START storage consumer in kv_client_stream_reconnect case >>>>>> + cd /tmp/tidb_cdc_test/kv_client_stream_reconnect + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/kv_client_stream_reconnect + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/kv_client_stream_reconnect/storage_test/kv-client-stream-reconnect-31843?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' wait process cdc.test exit for 1-th time... + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... wait process cdc.test exit for 2-th time... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 26 19:13:12 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.58645866.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + workdir=/tmp/tidb_cdc_test/canal_json_storage_basic + sink_uri='file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/canal_json_storage_basic ++ date + echo '[Fri Apr 26 19:13:13 CST 2024] <<<<<< START storage consumer in canal_json_storage_basic case >>>>>>' [Fri Apr 26 19:13:13 CST 2024] <<<<<< START storage consumer in canal_json_storage_basic case >>>>>> + cd /tmp/tidb_cdc_test/canal_json_storage_basic + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/canal_json_storage_basic + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/canal_json_storage_basic/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/canal_json_storage_basic/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_basic/conf/changefeed.toml ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:15 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/79b61d55-d752-4fc7-b06d-3201818fa24b {"id":"79b61d55-d752-4fc7-b06d-3201818fa24b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d232d 79b61d55-d752-4fc7-b06d-3201818fa24b /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/79b61d55-d752-4fc7-b06d-3201818fa24b {"id":"79b61d55-d752-4fc7-b06d-3201818fa24b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d232d 79b61d55-d752-4fc7-b06d-3201818fa24b /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/79b61d55-d752-4fc7-b06d-3201818fa24b {"id":"79b61d55-d752-4fc7-b06d-3201818fa24b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d232d 79b61d55-d752-4fc7-b06d-3201818fa24b /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5910.out cli changefeed create --start-ts=449348880645226497 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true' -c changefeed-error-2 Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7362132000486624707,"namespace":"default","id":"changefeed-error-2","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:13:16.345586281+08:00","start_ts":449348880645226497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348880645226497,"checkpoint_ts":449348880645226497,"checkpoint_time":"2024-04-26 19:12:26.309"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:13:17 CST 2024] <<<<<< run test case owner_resign success! >>>>>> + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + info='{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:13:16.359299637+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348880645226497 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:13:16.359299637+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348880645226497 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:13:16.359299637+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348880645226497 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-04-26T19:13:16.359299637+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348880645226497 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-04-26T19:13:16.359299637+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348880645226497 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348880645226497 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348880645226497 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5967.out cli changefeed remove -c changefeed-error-2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 0 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687042580014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4, pid:4122, start at 2024-04-26 19:13:18.006999999 +0800 CST m=+5.525718214 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:18.014 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:17.974 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:17.974 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687042580014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4, pid:4122, start at 2024-04-26 19:13:18.006999999 +0800 CST m=+5.525718214 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:18.014 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:17.974 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:17.974 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687045640003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-h0xr4, pid:4192, start at 2024-04-26 19:13:18.170486572 +0800 CST m=+5.617954920 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:18.176 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:18.169 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:18.169 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:13:21 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.60006002.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.5492.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6870810c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:6298, start at 2024-04-26 19:13:22.005682417 +0800 CST m=+5.521443754 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:22.017 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:21.987 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:21.987 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... + set +x + tso='449348895263686657 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348895263686657 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:13:23 CST 2024] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.55255527.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6870810c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:6298, start at 2024-04-26 19:13:22.005682417 +0800 CST m=+5.521443754 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:22.017 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:21.987 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:21.987 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687081dc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:6354, start at 2024-04-26 19:13:22.07172792 +0800 CST m=+5.504638419 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:22.079 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:22.039 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:22.039 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:24 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1965608d-ee62-4c2c-bb1f-f1b3050315c4 {"id":"1965608d-ee62-4c2c-bb1f-f1b3050315c4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d2369 1965608d-ee62-4c2c-bb1f-f1b3050315c4 /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1965608d-ee62-4c2c-bb1f-f1b3050315c4 {"id":"1965608d-ee62-4c2c-bb1f-f1b3050315c4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d2369 1965608d-ee62-4c2c-bb1f-f1b3050315c4 /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1965608d-ee62-4c2c-bb1f-f1b3050315c4 {"id":"1965608d-ee62-4c2c-bb1f-f1b3050315c4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b0d2369 1965608d-ee62-4c2c-bb1f-f1b3050315c4 /tidb/cdc/default/default/upstream/7362132000486624707 {"id":7362132000486624707,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6045.out cli changefeed create --start-ts=0 '--sink-uri=file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true' -c changefeed-initialize-error Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7362132000486624707,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:13:25.346714051+08:00","start_ts":449348896085770243,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348896085770243,"checkpoint_ts":449348896085770243,"checkpoint_time":"2024-04-26 19:13:25.210"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:26 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e445eb76-4918-4810-9543-b4c92ba960ef {"id":"e445eb76-4918-4810-9543-b4c92ba960ef","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1be91bf4 e445eb76-4918-4810-9543-b4c92ba960ef /tidb/cdc/default/default/upstream/7362132247474321160 {"id":7362132247474321160,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e445eb76-4918-4810-9543-b4c92ba960ef {"id":"e445eb76-4918-4810-9543-b4c92ba960ef","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1be91bf4 e445eb76-4918-4810-9543-b4c92ba960ef /tidb/cdc/default/default/upstream/7362132247474321160 {"id":7362132247474321160,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e445eb76-4918-4810-9543-b4c92ba960ef {"id":"e445eb76-4918-4810-9543-b4c92ba960ef","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1be91bf4 e445eb76-4918-4810-9543-b4c92ba960ef /tidb/cdc/default/default/upstream/7362132247474321160 {"id":7362132247474321160,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.5579.out cli changefeed create --start-ts=449348895263686657 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-3286?protocol=canal-json&enable-tidb-extension=true' + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s [Fri Apr 26 19:13:26 CST 2024] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.75807582.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 886ace9e-d06e-45cd-aa22-db9fb689e5bd Info: {"upstream_id":7362132247474321160,"namespace":"default","id":"886ace9e-d06e-45cd-aa22-db9fb689e5bd","sink_uri":"file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-3286?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:13:27.020486091+08:00","start_ts":449348895263686657,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348895263686657,"checkpoint_ts":449348895263686657,"checkpoint_time":"2024-04-26 19:13:22.074"} PASS + info='{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348896085770243, "checkpoint_time": "2024-04-26 19:13:25.210", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348896085770243, "checkpoint_time": "2024-04-26 19:13:25.210", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348896085770243, "checkpoint_time": "2024-04-26 19:13:25.210", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 449348896085770243, '"checkpoint_time":' '"2024-04-26' '19:13:25.210",' '"error":' '{' '"time":' '"2024-04-26T19:13:25.538803268+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later coverage: 2.5% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully [Fri Apr 26 19:13:25 CST 2024] <<<<<< run test case canal_json_storage_basic success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + workdir=/tmp/tidb_cdc_test/ddl_attributes + sink_uri='file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-3286?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_attributes ++ date + echo '[Fri Apr 26 19:13:28 CST 2024] <<<<<< START storage consumer in ddl_attributes case >>>>>>' [Fri Apr 26 19:13:28 CST 2024] <<<<<< START storage consumer in ddl_attributes case >>>>>> + cd /tmp/tidb_cdc_test/ddl_attributes + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_attributes + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_attributes/storage_test/ticdc-ddl-attributes-test-3286?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348896085770243, "checkpoint_time": "2024-04-26 19:13:25.210", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348896085770243, "checkpoint_time": "2024-04-26 19:13:25.210", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348896085770243, "checkpoint_time": "2024-04-26 19:13:25.210", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 449348896085770243, '"checkpoint_time":' '"2024-04-26' '19:13:25.210",' '"error":' '{' '"time":' '"2024-04-26T19:13:25.538803268+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:29 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c9aa655-5b94-4496-ba65-0a0bbc6c0d25 {"id":"1c9aa655-5b94-4496-ba65-0a0bbc6c0d25","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bfabbf3 1c9aa655-5b94-4496-ba65-0a0bbc6c0d25 /tidb/cdc/default/default/upstream/7362132262340521109 {"id":7362132262340521109,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c9aa655-5b94-4496-ba65-0a0bbc6c0d25 {"id":"1c9aa655-5b94-4496-ba65-0a0bbc6c0d25","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bfabbf3 1c9aa655-5b94-4496-ba65-0a0bbc6c0d25 /tidb/cdc/default/default/upstream/7362132262340521109 {"id":7362132262340521109,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c9aa655-5b94-4496-ba65-0a0bbc6c0d25 {"id":"1c9aa655-5b94-4496-ba65-0a0bbc6c0d25","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bfabbf3 1c9aa655-5b94-4496-ba65-0a0bbc6c0d25 /tidb/cdc/default/default/upstream/7362132262340521109 + grep -q 'etcd info' {"id":7362132262340521109,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x + workdir=/tmp/tidb_cdc_test/processor_stop_delay + sink_uri='file:///tmp/tidb_cdc_test/processor_stop_delay/storage_test/ticdc-processor-stop-delay-19894?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_stop_delay ++ date + echo '[Fri Apr 26 19:13:30 CST 2024] <<<<<< START storage consumer in processor_stop_delay case >>>>>>' [Fri Apr 26 19:13:30 CST 2024] <<<<<< START storage consumer in processor_stop_delay case >>>>>> + cd /tmp/tidb_cdc_test/processor_stop_delay + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_stop_delay + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_stop_delay/storage_test/ticdc-processor-stop-delay-19894?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table processor_stop_delay.t not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6870f154000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8, pid:3860, start at 2024-04-26 19:13:29.189766325 +0800 CST m=+33.770398907 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:29.200 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:29.173 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:29.173 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6870f154000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8, pid:3860, start at 2024-04-26 19:13:29.189766325 +0800 CST m=+33.770398907 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:29.200 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:29.173 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:29.173 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f53280014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kvjx8, pid:3928, start at 2024-04-26 19:13:02.716477665 +0800 CST m=+7.229781299 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:02.724 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:02.716 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:02.716 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68710d8c000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:7478, start at 2024-04-26 19:13:30.991632299 +0800 CST m=+5.378566997 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:30.998 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:30.979 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:30.979 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68710d8c000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:7478, start at 2024-04-26 19:13:30.991632299 +0800 CST m=+5.378566997 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:30.998 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:30.979 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:30.979 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68710d940013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:7570, start at 2024-04-26 19:13:31.00493801 +0800 CST m=+5.337847150 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:31.011 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:30.981 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:30.981 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Fri Apr 26 19:13:30 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ true != \n\o ]] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.53665368.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None Verifying downstream PD is started... table processor_stop_delay.t not exists for 2-th check, retry later check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": null }' + echo '{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": null }' { "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": null } ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449348898064957444, '"checkpoint_time":' '"2024-04-26' '19:13:32.760",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449348898064957444, '"checkpoint_time":' '"2024-04-26' '19:13:32.760",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6188.out cli changefeed pause -c changefeed-initialize-error PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Fri Apr 26 19:13:34 CST 2024] <<<<<< START cdc server in csv_storage_multi_tables_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_multi_tables_ddl.88558857.out server --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 26 19:13:34 CST 2024] <<<<<< START cdc server in multi_rocks case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.1674416746.out server --log-file /tmp/tidb_cdc_test/multi_rocks/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_rocks/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table processor_stop_delay.t not exists for 3-th check, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped changefeed new redo manager injected error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=changefeed + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": { "time": "2024-04-26T19:13:25.538803268+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 449348898064957444, '"checkpoint_time":' '"2024-04-26' '19:13:32.760",' '"error":' '{' '"time":' '"2024-04-26T19:13:25.538803268+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 449348898064957444, '"checkpoint_time":' '"2024-04-26' '19:13:32.760",' '"error":' '{' '"time":' '"2024-04-26T19:13:25.538803268+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .error.message + message='changefeed new redo manager injected error' + [[ ! changefeed new redo manager injected error =~ changefeed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6251.out cli changefeed resume -c changefeed-initialize-error =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... table processor_stop_delay.t not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:37 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d746ec5-c058-4717-bf7f-e45099cd9de9 {"id":"2d746ec5-c058-4717-bf7f-e45099cd9de9","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba68024 2d746ec5-c058-4717-bf7f-e45099cd9de9 /tidb/cdc/default/default/upstream/7362132172002487414 {"id":7362132172002487414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d746ec5-c058-4717-bf7f-e45099cd9de9 {"id":"2d746ec5-c058-4717-bf7f-e45099cd9de9","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba68024 2d746ec5-c058-4717-bf7f-e45099cd9de9 /tidb/cdc/default/default/upstream/7362132172002487414 {"id":7362132172002487414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d746ec5-c058-4717-bf7f-e45099cd9de9 {"id":"2d746ec5-c058-4717-bf7f-e45099cd9de9","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba68024 2d746ec5-c058-4717-bf7f-e45099cd9de9 /tidb/cdc/default/default/upstream/7362132172002487414 {"id":7362132172002487414,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.cli.17110.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-14655?protocol=canal-json&enable-tidb-extension=true' + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s Create changefeed successfully! ID: 2e709cf7-9332-43b9-9c64-1db2bc5fe0d7 Info: {"upstream_id":7362132172002487414,"namespace":"default","id":"2e709cf7-9332-43b9-9c64-1db2bc5fe0d7","sink_uri":"file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-14655?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:13:37.493865818+08:00","start_ts":449348899287072769,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348899287072769,"checkpoint_ts":449348899287072769,"checkpoint_time":"2024-04-26 19:13:37.422"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + info='{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": null }' + echo '{ "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": null }' { "upstream_id": 7362132000486624707, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348898064957444, "checkpoint_time": "2024-04-26 19:13:32.760", "error": null } ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449348898064957444, '"checkpoint_time":' '"2024-04-26' '19:13:32.760",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132000486624707, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449348898064957444, '"checkpoint_time":' '"2024-04-26' '19:13:32.760",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6321.out cli changefeed remove -c changefeed-initialize-error + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:37 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5eaaaa85-14f6-4caf-b1af-829b975e65c1 {"id":"5eaaaa85-14f6-4caf-b1af-829b975e65c1","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c1c4ef2 5eaaaa85-14f6-4caf-b1af-829b975e65c1 /tidb/cdc/default/default/upstream/7362132299464565496 {"id":7362132299464565496,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5eaaaa85-14f6-4caf-b1af-829b975e65c1 {"id":"5eaaaa85-14f6-4caf-b1af-829b975e65c1","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c1c4ef2 5eaaaa85-14f6-4caf-b1af-829b975e65c1 /tidb/cdc/default/default/upstream/7362132299464565496 {"id":7362132299464565496,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5eaaaa85-14f6-4caf-b1af-829b975e65c1 {"id":"5eaaaa85-14f6-4caf-b1af-829b975e65c1","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c1c4ef2 5eaaaa85-14f6-4caf-b1af-829b975e65c1 /tidb/cdc/default/default/upstream/7362132299464565496 {"id":7362132299464565496,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: test-normal Info: {"upstream_id":7362132299464565496,"namespace":"default","id":"test-normal","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-12850?flush-interval=5s\u0026protocol=csv","create_time":"2024-04-26T19:13:37.67724136+08:00","start_ts":449348898489368577,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t1","multi_tables_ddl_test.t2","multi_tables_ddl_test.t3","multi_tables_ddl_test.t4","multi_tables_ddl_test.t1_7","multi_tables_ddl_test.t2_7","multi_tables_ddl_test.finish_mark"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348898489368577,"checkpoint_ts":449348898489368577,"checkpoint_time":"2024-04-26 19:13:34.379"} Create changefeed successfully! ID: test-error-1 Info: {"upstream_id":7362132299464565496,"namespace":"default","id":"test-error-1","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-5670?flush-interval=5s\u0026protocol=csv","create_time":"2024-04-26T19:13:37.826915727+08:00","start_ts":449348898489368577,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t5","multi_tables_ddl_test.t6","multi_tables_ddl_test.t7","multi_tables_ddl_test.t8"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348898489368577,"checkpoint_ts":449348898489368577,"checkpoint_time":"2024-04-26 19:13:34.379"} Create changefeed successfully! ID: test-error-2 Info: {"upstream_id":7362132299464565496,"namespace":"default","id":"test-error-2","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-13466?flush-interval=5s\u0026protocol=csv","create_time":"2024-04-26T19:13:37.977445691+08:00","start_ts":449348898489368577,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t9","multi_tables_ddl_test.t10"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348898489368577,"checkpoint_ts":449348898489368577,"checkpoint_time":"2024-04-26 19:13:34.379"} + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-12850?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml + log_suffix=1 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Fri Apr 26 19:13:37 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Fri Apr 26 19:13:37 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer1.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-normal-12850?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/normal.toml + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-5670?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml + log_suffix=2 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Fri Apr 26 19:13:38 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Fri Apr 26 19:13:38 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer2.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-1-5670?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-1.toml + workdir=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-13466?flush-interval=5s&protocol=csv' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml + log_suffix=3 ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_multi_tables_ddl ++ date + echo '[Fri Apr 26 19:13:38 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>>' [Fri Apr 26 19:13:38 CST 2024] <<<<<< START storage consumer in csv_storage_multi_tables_ddl case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/cdc_storage_consumer3.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_multi_tables_ddl/storage_test/ticdc-multi-tables-ddl-test-error-2-13466?flush-interval=5s&protocol=csv' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/conf/error-2.toml start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 449348898064957444 SinkURI: file:///tmp/tidb_cdc_test/changefeed_error/storage_test/ticdc-sink-retry-test-27196?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... table processor_stop_delay.t not exists for 5-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/multi_rocks + sink_uri='file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-14655?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/multi_rocks ++ date + echo '[Fri Apr 26 19:13:38 CST 2024] <<<<<< START storage consumer in multi_rocks case >>>>>>' [Fri Apr 26 19:13:38 CST 2024] <<<<<< START storage consumer in multi_rocks case >>>>>> + cd /tmp/tidb_cdc_test/multi_rocks + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/multi_rocks + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/multi_rocks/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/multi_rocks/storage_test/ticdc-multi-rocks-test-14655?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... ***************** properties ***************** "readallfields"="true" "mysql.db"="multi_rocks" "updateproportion"="0" "recordcount"="1000" "mysql.port"="4000" "threadcount"="2" "insertproportion"="0" "operationcount"="0" "workload"="core" "mysql.host"="127.0.0.1" "readproportion"="0" "mysql.user"="root" "requestdistribution"="uniform" "table"="a1" "dotransactions"="false" "scanproportion"="0" ********************************************** table processor_stop_delay.t exists wait process cdc.test exit for 1-th time... check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... Run finished, takes 946.8118ms INSERT - Takes(s): 0.9, Count: 1000, OPS: 1069.8, Avg(us): 1854, Min(us): 863, Max(us): 87016, 95th(us): 3000, 99th(us): 5000 ***************** properties ***************** "mysql.db"="multi_rocks" "threadcount"="2" "recordcount"="1000" "operationcount"="0" "mysql.port"="4000" "table"="a2" "insertproportion"="0" "readallfields"="true" "mysql.host"="127.0.0.1" "mysql.user"="root" "dotransactions"="false" "requestdistribution"="uniform" "scanproportion"="0" "readproportion"="0" "updateproportion"="0" "workload"="core" ********************************************** cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:13:41 CST 2024] <<<<<< run test case changefeed_error success! >>>>>> table multi_tables_ddl_test.t55 not exists for 1-th check, retry later check diff failed 1-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Run finished, takes 1.285956908s INSERT - Takes(s): 1.1, Count: 1000, OPS: 931.3, Avg(us): 2539, Min(us): 836, Max(us): 212104, 95th(us): 3000, 99th(us): 64000 ***************** properties ***************** "scanproportion"="0" "threadcount"="2" "table"="a3" "operationcount"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "dotransactions"="false" "insertproportion"="0" "recordcount"="1000" "readproportion"="0" "updateproportion"="0" "requestdistribution"="uniform" "readallfields"="true" "mysql.db"="multi_rocks" "workload"="core" "mysql.user"="root" ********************************************** table multi_tables_ddl_test.t55 not exists for 2-th check, retry later check diff failed 1-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 32.13 secs (104729078 bytes/sec) [Pipeline] { [Pipeline] cache Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Run finished, takes 726.931085ms INSERT - Takes(s): 0.7, Count: 1000, OPS: 1417.4, Avg(us): 1416, Min(us): 832, Max(us): 77495, 95th(us): 2000, 99th(us): 3000 check diff failed 2-th time, retry later ***************** properties ***************** "mysql.port"="4000" "mysql.db"="multi_rocks" "dotransactions"="false" "mysql.user"="root" "scanproportion"="0" "threadcount"="2" "readallfields"="true" "workload"="core" "readproportion"="0" "insertproportion"="0" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "operationcount"="0" "table"="a4" "updateproportion"="0" "recordcount"="1000" ********************************************** VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6871b9700012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:3765, start at 2024-04-26 19:13:42.006195181 +0800 CST m=+5.394177032 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:42.013 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:41.980 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:41.980 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6871b9700012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:3765, start at 2024-04-26 19:13:42.006195181 +0800 CST m=+5.394177032 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:42.013 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:41.980 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:41.980 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6871bb040013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:3835, start at 2024-04-26 19:13:42.110706428 +0800 CST m=+5.437643678 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:42.118 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:42.081 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:42.081 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Run finished, takes 755.015847ms INSERT - Takes(s): 0.7, Count: 1000, OPS: 1355.9, Avg(us): 1475, Min(us): 789, Max(us): 74557, 95th(us): 2000, 99th(us): 4000 table multi_tables_ddl_test.t55 not exists for 3-th check, retry later ***************** properties ***************** "updateproportion"="0" "readallfields"="true" "readproportion"="0" "threadcount"="2" "recordcount"="1000" "operationcount"="0" "workload"="core" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "mysql.db"="multi_rocks" "scanproportion"="0" "mysql.user"="root" "dotransactions"="false" "mysql.port"="4000" "insertproportion"="0" "table"="a5" ********************************************** check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.attributes_t1_new not exists for 1-th check, retry later Run finished, takes 707.347276ms INSERT - Takes(s): 0.7, Count: 1000, OPS: 1534.2, Avg(us): 1381, Min(us): 815, Max(us): 56491, 95th(us): 2000, 99th(us): 4000 check diff failed 3-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5148.out cli tso query --pd=http://127.0.0.1:2379 table multi_tables_ddl_test.t55 not exists for 4-th check, retry later check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/canal_json_storage_partition_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.attributes_t1_new not exists for 2-th check, retry later table multi_rocks.finish_mark not exists for 1-th check, retry later check diff failed 4-th time, retry later + set +x + tso='449348901884395524 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348901884395524 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x table multi_tables_ddl_test.t55 not exists for 5-th check, retry later [Fri Apr 26 19:13:49 CST 2024] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.52035205.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later table multi_rocks.finish_mark not exists for 2-th check, retry later check diff failed 4-th time, retry later invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Fri Apr 26 19:13:50 CST 2024] <<<<<< run all test cases cdc_server_tips success! >>>>>> check diff failed 5-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table multi_tables_ddl_test.t55 not exists for 6-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68722b640009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:4366, start at 2024-04-26 19:13:49.28189151 +0800 CST m=+5.354003773 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:49.292 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:49.273 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:49.273 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68722b640009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:4366, start at 2024-04-26 19:13:49.28189151 +0800 CST m=+5.354003773 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:49.292 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:49.273 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:49.273 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68722dac0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:4434, start at 2024-04-26 19:13:49.446997033 +0800 CST m=+5.420531091 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:49.455 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:49.419 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:49.419 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table ddl_attributes.finish_mark not exists for 2-th check, retry later table multi_rocks.finish_mark not exists for 3-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 5-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:52 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e069f0a7-cd72-45de-8096-3caa5170942f {"id":"e069f0a7-cd72-45de-8096-3caa5170942f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c43c30e e069f0a7-cd72-45de-8096-3caa5170942f /tidb/cdc/default/default/upstream/7362132339111663868 {"id":7362132339111663868,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e069f0a7-cd72-45de-8096-3caa5170942f {"id":"e069f0a7-cd72-45de-8096-3caa5170942f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c43c30e e069f0a7-cd72-45de-8096-3caa5170942f /tidb/cdc/default/default/upstream/7362132339111663868 {"id":7362132339111663868,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e069f0a7-cd72-45de-8096-3caa5170942f {"id":"e069f0a7-cd72-45de-8096-3caa5170942f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c43c30e e069f0a7-cd72-45de-8096-3caa5170942f /tidb/cdc/default/default/upstream/7362132339111663868 {"id":7362132339111663868,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.5251.out cli changefeed create --start-ts=449348901884395524 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-13445?protocol=canal-json&enable-tidb-extension=true' check diff failed 6-th time, retry later Create changefeed successfully! ID: 0ea26f60-044c-4f5a-b578-188332918d9e Info: {"upstream_id":7362132339111663868,"namespace":"default","id":"0ea26f60-044c-4f5a-b578-188332918d9e","sink_uri":"file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-13445?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:13:53.354104418+08:00","start_ts":449348901884395524,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348901884395524,"checkpoint_ts":449348901884395524,"checkpoint_time":"2024-04-26 19:13:47.330"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table multi_tables_ddl_test.t55 not exists for 7-th check, retry later table multi_rocks.finish_mark not exists for 4-th check, retry later [Fri Apr 26 19:13:54 CST 2024] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.56965698.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= check diff successfully + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + workdir=/tmp/tidb_cdc_test/ddl_puller_lag + sink_uri='file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-13445?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_puller_lag ++ date + echo '[Fri Apr 26 19:13:54 CST 2024] <<<<<< START storage consumer in ddl_puller_lag case >>>>>>' [Fri Apr 26 19:13:54 CST 2024] <<<<<< START storage consumer in ddl_puller_lag case >>>>>> + cd /tmp/tidb_cdc_test/ddl_puller_lag + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_puller_lag + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_puller_lag/storage_test/ticdc-ddl-puller-lag-test-13445?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table ddl_attributes.finish_mark not exists for 3-th check, retry later wait process cdc.test exit for 1-th time... check diff failed 7-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:13:55 CST 2024] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> table multi_tables_ddl_test.t55 not exists for 8-th check, retry later table ddl_attributes.finish_mark not exists for 4-th check, retry later table multi_rocks.finish_mark not exists for 5-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 10.28 secs (327291090 bytes/sec) [Pipeline] { [Pipeline] cache check diff failed 8-th time, retry later Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:57 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c52c9cdc-450a-4577-a7d6-97a053a93a11 {"id":"c52c9cdc-450a-4577-a7d6-97a053a93a11","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c600bf3 c52c9cdc-450a-4577-a7d6-97a053a93a11 /tidb/cdc/default/default/upstream/7362132380169028022 {"id":7362132380169028022,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c52c9cdc-450a-4577-a7d6-97a053a93a11 {"id":"c52c9cdc-450a-4577-a7d6-97a053a93a11","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c600bf3 c52c9cdc-450a-4577-a7d6-97a053a93a11 /tidb/cdc/default/default/upstream/7362132380169028022 {"id":7362132380169028022,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c52c9cdc-450a-4577-a7d6-97a053a93a11 {"id":"c52c9cdc-450a-4577-a7d6-97a053a93a11","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c600bf3 c52c9cdc-450a-4577-a7d6-97a053a93a11 /tidb/cdc/default/default/upstream/7362132380169028022 {"id":7362132380169028022,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/processor_etcd_worker_delay + sink_uri='file:///tmp/tidb_cdc_test/processor_etcd_worker_delay/storage_test/processor-etcd-delay-9050?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_etcd_worker_delay ++ date + echo '[Fri Apr 26 19:13:57 CST 2024] <<<<<< START storage consumer in processor_etcd_worker_delay case >>>>>>' [Fri Apr 26 19:13:57 CST 2024] <<<<<< START storage consumer in processor_etcd_worker_delay case >>>>>> + cd /tmp/tidb_cdc_test/processor_etcd_worker_delay + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_etcd_worker_delay + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_etcd_worker_delay/storage_test/processor-etcd-delay-9050?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table ddl_attributes.finish_mark not exists for 5-th check, retry later table multi_rocks.finish_mark not exists for 6-th check, retry later check diff failed 9-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_tables_ddl_test.t55 exists table multi_tables_ddl_test.t66 exists table multi_tables_ddl_test.t7 exists table multi_tables_ddl_test.t88 exists table multi_tables_ddl_test.finish_mark not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 6-th check, retry later table multi_rocks.finish_mark not exists for 7-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6872ced8000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9, pid:4598, start at 2024-04-26 19:13:59.750876234 +0800 CST m=+6.864822637 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:59.758 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:59.734 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:59.734 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6872ced8000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9, pid:4598, start at 2024-04-26 19:13:59.750876234 +0800 CST m=+6.864822637 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:59.758 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:59.734 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:59.734 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6872b7600013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-8jrk9, pid:4669, start at 2024-04-26 19:13:58.259022222 +0800 CST m=+5.320860606 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:58.266 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:58.232 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:58.232 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/canal_json_storage_partition_table/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table multi_tables_ddl_test.finish_mark not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table ddl_attributes.finish_mark not exists for 7-th check, retry later table multi_rocks.finish_mark not exists for 8-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:14:02 CST 2024] <<<<<< run test case processor_stop_delay success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table multi_tables_ddl_test.finish_mark not exists for 3-th check, retry later [Fri Apr 26 19:14:03 CST 2024] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.59215923.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc0.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc0 --cluster-id default --addr 127.0.0.1:8300 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_rocks.finish_mark not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 8-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table multi_tables_ddl_test.finish_mark not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687326c0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:7290, start at 2024-04-26 19:14:05.377159691 +0800 CST m=+5.248947382 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:05.384 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:05.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:05.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 9-th check, retry later table multi_rocks.finish_mark not exists for 10-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:06 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:14:06 CST 2024] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.59725974.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_tables_ddl_test.finish_mark not exists for 5-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:14:07 CST 2024] <<<<<< run test case cdc success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687326c0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:7290, start at 2024-04-26 19:14:05.377159691 +0800 CST m=+5.248947382 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:05.384 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:05.360 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:05.360 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687328e80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:7361, start at 2024-04-26 19:14:05.531256913 +0800 CST m=+5.343016159 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:05.539 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:05.498 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:05.498 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_attributes.finish_mark not exists for 10-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table multi_rocks.finish_mark not exists for 11-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:09 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e7c9863-c437-4715-86d8-648106e3f899 {"id":"8e7c9863-c437-4715-86d8-648106e3f899","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f819 8e7c9863-c437-4715-86d8-648106e3f899 /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e7c9863-c437-4715-86d8-648106e3f899 {"id":"8e7c9863-c437-4715-86d8-648106e3f899","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f819 8e7c9863-c437-4715-86d8-648106e3f899 /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e7c9863-c437-4715-86d8-648106e3f899 {"id":"8e7c9863-c437-4715-86d8-648106e3f899","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f819 8e7c9863-c437-4715-86d8-648106e3f899 /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:14:09 CST 2024] <<<<<< START cdc server in canal_json_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.60166018.out server --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdccdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_datacdc2 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table multi_tables_ddl_test.finish_mark exists check table exists success + endpoints=http://127.0.0.1:2379 + changefeed_id=test-normal + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-normal -s + info='{ "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449348907494014980, "checkpoint_time": "2024-04-26 19:14:08.729", "error": null }' + echo '{ "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449348907494014980, "checkpoint_time": "2024-04-26 19:14:08.729", "error": null }' { "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449348907494014980, "checkpoint_time": "2024-04-26 19:14:08.729", "error": null } ++ echo '{' '"upstream_id":' 7362132299464565496, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 449348907494014980, '"checkpoint_time":' '"2024-04-26' '19:14:08.729",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132299464565496, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 449348907494014980, '"checkpoint_time":' '"2024-04-26' '19:14:08.729",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-1 + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-1 -s + info='{ "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449348907494014980, "checkpoint_time": "2024-04-26 19:14:08.729", "error": null }' + echo '{ "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449348907494014980, "checkpoint_time": "2024-04-26 19:14:08.729", "error": null }' { "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449348907494014980, "checkpoint_time": "2024-04-26 19:14:08.729", "error": null } ++ echo '{' '"upstream_id":' 7362132299464565496, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 449348907494014980, '"checkpoint_time":' '"2024-04-26' '19:14:08.729",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132299464565496, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 449348907494014980, '"checkpoint_time":' '"2024-04-26' '19:14:08.729",' '"error":' null '}' ++ jq -r .error.message =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-2 + expected_state=failed + error_msg=ErrSyncRenameTableFailed + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-2 -s + info='{ "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449348899787243538, "checkpoint_time": "2024-04-26 19:13:39.330", "error": { "time": "2024-04-26T19:13:41.760056014+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrHandleDDLFailed", "message": "[CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:13:41.33 +0800 CST, Err:\u003cnil\u003e, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348900311531527. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348900311531527]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' + echo '{ "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449348899787243538, "checkpoint_time": "2024-04-26 19:13:39.330", "error": { "time": "2024-04-26T19:13:41.760056014+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrHandleDDLFailed", "message": "[CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:13:41.33 +0800 CST, Err:\u003cnil\u003e, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348900311531527. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348900311531527]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' { "upstream_id": 7362132299464565496, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449348899787243538, "checkpoint_time": "2024-04-26 19:13:39.330", "error": { "time": "2024-04-26T19:13:41.760056014+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrHandleDDLFailed", "message": "[CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:13:41.33 +0800 CST, Err:\u003cnil\u003e, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348900311531527. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348900311531527]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '128', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7362132299464565496, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449348899787243538, '"checkpoint_time":' '"2024-04-26' '19:13:39.330",' '"error":' '{' '"time":' '"2024-04-26T19:13:41.760056014+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrHandleDDLFailed",' '"message":' '"[CDC:ErrHandleDDLFailed]handle' ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:13:41.33 +0800 CST, 'Err:\u003cnil\u003e,' ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348900311531527. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add '`ignore-txn-start-ts=[449348900311531527]`' to the changefeed in the filter configuration.: '[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''128'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7362132299464565496, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449348899787243538, '"checkpoint_time":' '"2024-04-26' '19:13:39.330",' '"error":' '{' '"time":' '"2024-04-26T19:13:41.760056014+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrHandleDDLFailed",' '"message":' '"[CDC:ErrHandleDDLFailed]handle' ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:13:41.33 +0800 CST, 'Err:\u003cnil\u003e,' ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348900311531527. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add '`ignore-txn-start-ts=[449348900311531527]`' to the changefeed in the filter configuration.: '[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''128'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' table ddl_attributes.finish_mark exists Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 11.37 secs (296022404 bytes/sec) [Pipeline] { [Pipeline] cache + message='[CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:13:41.33 +0800 CST, Err:, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348900311531527. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348900311531527]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule.' + [[ ! [CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:13:41.33 +0800 CST, Err:, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348900311531527. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348900311531527]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '128', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule. =~ ErrSyncRenameTableFailed ]] check diff successfully Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark exists check diff successfully check diff successfully + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8600.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:14:12 CST 2024] <<<<<< run test case multi_rocks success! >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:14:11 CST 2024] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> + set +x + tso='449348908026429441 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348908026429441 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:14:12 CST 2024] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.86348636.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 26 19:14:12 CST 2024] <<<<<< run test case ddl_attributes success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:12 GMT < Content-Length: 1144 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7836714a-85f9-4dea-97bc-6a9a199c7e84 {"id":"7836714a-85f9-4dea-97bc-6a9a199c7e84","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/8e7c9863-c437-4715-86d8-648106e3f899 {"id":"8e7c9863-c437-4715-86d8-648106e3f899","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f819 8e7c9863-c437-4715-86d8-648106e3f899 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f827 7836714a-85f9-4dea-97bc-6a9a199c7e84 /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7836714a-85f9-4dea-97bc-6a9a199c7e84 {"id":"7836714a-85f9-4dea-97bc-6a9a199c7e84","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/8e7c9863-c437-4715-86d8-648106e3f899 {"id":"8e7c9863-c437-4715-86d8-648106e3f899","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f819 8e7c9863-c437-4715-86d8-648106e3f899 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f827 7836714a-85f9-4dea-97bc-6a9a199c7e84 /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7836714a-85f9-4dea-97bc-6a9a199c7e84 {"id":"7836714a-85f9-4dea-97bc-6a9a199c7e84","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/8e7c9863-c437-4715-86d8-648106e3f899 {"id":"8e7c9863-c437-4715-86d8-648106e3f899","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b3e872b1-88c2-4030-b9a7-33b7390e6ccc {"id":"b3e872b1-88c2-4030-b9a7-33b7390e6ccc","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f7f2 b3e872b1-88c2-4030-b9a7-33b7390e6ccc /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f819 8e7c9863-c437-4715-86d8-648106e3f899 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c86f827 7836714a-85f9-4dea-97bc-6a9a199c7e84 /tidb/cdc/default/default/upstream/7362132418065225981 {"id":7362132418065225981,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.canal_json_storage_partition_table.cli.6068.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Create changefeed successfully! ID: eb1f452d-8ca7-46b8-9be2-c54e91237370 Info: {"upstream_id":7362132418065225981,"namespace":"default","id":"eb1f452d-8ca7-46b8-9be2-c54e91237370","sink_uri":"file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:14:15.101727427+08:00","start_ts":449348909133725711,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348909133725711,"checkpoint_ts":449348909133725711,"checkpoint_time":"2024-04-26 19:14:14.984"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:15 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b88bd1-0dbe-4cbd-954a-342702ea9f69 {"id":"66b88bd1-0dbe-4cbd-954a-342702ea9f69","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ca32ff3 66b88bd1-0dbe-4cbd-954a-342702ea9f69 /tidb/cdc/default/default/upstream/7362132447298295763 {"id":7362132447298295763,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b88bd1-0dbe-4cbd-954a-342702ea9f69 {"id":"66b88bd1-0dbe-4cbd-954a-342702ea9f69","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ca32ff3 66b88bd1-0dbe-4cbd-954a-342702ea9f69 /tidb/cdc/default/default/upstream/7362132447298295763 {"id":7362132447298295763,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66b88bd1-0dbe-4cbd-954a-342702ea9f69 {"id":"66b88bd1-0dbe-4cbd-954a-342702ea9f69","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ca32ff3 66b88bd1-0dbe-4cbd-954a-342702ea9f69 /tidb/cdc/default/default/upstream/7362132447298295763 {"id":7362132447298295763,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8681.out cli changefeed create --start-ts=449348908026429441 '--sink-uri=file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-761?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: d11f9f58-de14-433f-88d8-a1ce88fd6fcb Info: {"upstream_id":7362132447298295763,"namespace":"default","id":"d11f9f58-de14-433f-88d8-a1ce88fd6fcb","sink_uri":"file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-761?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:14:15.625364594+08:00","start_ts":449348908026429441,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348908026429441,"checkpoint_ts":449348908026429441,"checkpoint_time":"2024-04-26 19:14:10.760"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6873c6400013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:6301, start at 2024-04-26 19:14:15.618430848 +0800 CST m=+5.401514023 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:15.628 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:15.618 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:15.618 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x + set +x + workdir=/tmp/tidb_cdc_test/ddl_sequence + sink_uri='file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-761?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_sequence ++ date + echo '[Fri Apr 26 19:14:17 CST 2024] <<<<<< START storage consumer in ddl_sequence case >>>>>>' [Fri Apr 26 19:14:17 CST 2024] <<<<<< START storage consumer in ddl_sequence case >>>>>> + cd /tmp/tidb_cdc_test/ddl_sequence + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_sequence + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_sequence/storage_test/ticdc-ddl-sequence-test-761?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6873c6400013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:6301, start at 2024-04-26 19:14:15.618430848 +0800 CST m=+5.401514023 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:15.628 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:15.618 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:15.618 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6873c7c4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-wnxvf, pid:6377, start at 2024-04-26 19:14:15.68703052 +0800 CST m=+5.397093007 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:15.706 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:15.665 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:15.665 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_sequence.finish_mark not exists for 1-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 6.47 secs (519895697 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:14:20 CST 2024] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.76767678.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_sequence.finish_mark not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_sequence.finish_mark not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:23 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9f65c754-bbf6-42aa-92f0-ff03dd252e84 {"id":"9f65c754-bbf6-42aa-92f0-ff03dd252e84","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cc8b8f2 9f65c754-bbf6-42aa-92f0-ff03dd252e84 /tidb/cdc/default/default/upstream/7362132496661233983 {"id":7362132496661233983,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9f65c754-bbf6-42aa-92f0-ff03dd252e84 {"id":"9f65c754-bbf6-42aa-92f0-ff03dd252e84","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cc8b8f2 9f65c754-bbf6-42aa-92f0-ff03dd252e84 /tidb/cdc/default/default/upstream/7362132496661233983 {"id":7362132496661233983,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9f65c754-bbf6-42aa-92f0-ff03dd252e84 {"id":"9f65c754-bbf6-42aa-92f0-ff03dd252e84","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cc8b8f2 9f65c754-bbf6-42aa-92f0-ff03dd252e84 /tidb/cdc/default/default/upstream/7362132496661233983 {"id":7362132496661233983,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.7723.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-25600?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml \033[0;36m<<< Run all test success >>>\033[0m Create changefeed successfully! ID: 08f5084e-1a42-4a1e-bb58-a2650b91912d Info: {"upstream_id":7362132496661233983,"namespace":"default","id":"08f5084e-1a42-4a1e-bb58-a2650b91912d","sink_uri":"file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-25600?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:14:24.067986103+08:00","start_ts":449348911488827396,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348911488827396,"checkpoint_ts":449348911488827396,"checkpoint_time":"2024-04-26 19:14:23.968"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... \033[0;36m<<< Run all test success >>>\033[0m VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68744220000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9, pid:4257, start at 2024-04-26 19:14:23.508048652 +0800 CST m=+5.218615101 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:23.515 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:23.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:23.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68744220000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9, pid:4257, start at 2024-04-26 19:14:23.508048652 +0800 CST m=+5.218615101 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:23.515 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:23.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:23.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687442cc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-k03r9, pid:4328, start at 2024-04-26 19:14:23.581532054 +0800 CST m=+5.215793612 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:23.588 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:23.589 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:23.589 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_sequence.finish_mark not exists for 4-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/event_filter + sink_uri='file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-25600?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/event_filter ++ date + echo '[Fri Apr 26 19:14:25 CST 2024] <<<<<< START storage consumer in event_filter case >>>>>>' [Fri Apr 26 19:14:25 CST 2024] <<<<<< START storage consumer in event_filter case >>>>>> + cd /tmp/tidb_cdc_test/event_filter + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/event_filter + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/event_filter/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/event_filter/storage_test/ticdc-event-filter-25600?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table event_filter.t1 does not exists table event_filter.t1 not exists for 1-th check, retry later + workdir=/tmp/tidb_cdc_test/canal_json_storage_partition_table + sink_uri='file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/canal_json_storage_partition_table ++ date + echo '[Fri Apr 26 19:14:26 CST 2024] <<<<<< START storage consumer in canal_json_storage_partition_table case >>>>>>' [Fri Apr 26 19:14:26 CST 2024] <<<<<< START storage consumer in canal_json_storage_partition_table case >>>>>> + cd /tmp/tidb_cdc_test/canal_json_storage_partition_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/canal_json_storage_partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/canal_json_storage_partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/canal_json_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/conf/changefeed.toml + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.5568.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_sequence.finish_mark not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table event_filter.t1 not exists for 2-th check, retry later + set +x + tso='449348912256647169 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348912256647169 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x table ddl_sequence.finish_mark not exists for 6-th check, retry later table processor_delay.t1 exists table processor_delay.t2 exists table processor_delay.t3 exists table processor_delay.t4 not exists for 1-th check, retry later table processor_delay.t4 exists table processor_delay.t5 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687477ec0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:9157, start at 2024-04-26 19:14:26.960483593 +0800 CST m=+5.291738687 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:26.968 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:26.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:26.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687477ec0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:9157, start at 2024-04-26 19:14:26.960483593 +0800 CST m=+5.291738687 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:26.968 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:26.939 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:26.939 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687478a80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3gkpk, pid:9233, start at 2024-04-26 19:14:27.00841109 +0800 CST m=+5.289365244 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:27.015 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:26.986 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:26.986 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 26 19:14:29 CST 2024] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.56015603.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table event_filter.t1 not exists for 3-th check, retry later table processor_delay.t5 exists table processor_delay.t6 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 7-th check, retry later table event_filter.t1 not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/csv_storage_partition_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table processor_delay.t6 not exists for 2-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10484.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:32 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c035efab-7799-4767-89a1-7d206618f050 {"id":"c035efab-7799-4767-89a1-7d206618f050","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ce84815 c035efab-7799-4767-89a1-7d206618f050 /tidb/cdc/default/default/upstream/7362132527842577306 {"id":7362132527842577306,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c035efab-7799-4767-89a1-7d206618f050 {"id":"c035efab-7799-4767-89a1-7d206618f050","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ce84815 c035efab-7799-4767-89a1-7d206618f050 /tidb/cdc/default/default/upstream/7362132527842577306 {"id":7362132527842577306,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c035efab-7799-4767-89a1-7d206618f050 {"id":"c035efab-7799-4767-89a1-7d206618f050","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ce84815 c035efab-7799-4767-89a1-7d206618f050 /tidb/cdc/default/default/upstream/7362132527842577306 {"id":7362132527842577306,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.5648.out cli changefeed create --start-ts=449348912256647169 '--sink-uri=file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-25640?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 9b0537dd-862c-4617-b7b3-d75f6998d4c0 Info: {"upstream_id":7362132527842577306,"namespace":"default","id":"9b0537dd-862c-4617-b7b3-d75f6998d4c0","sink_uri":"file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-25640?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:14:33.345163035+08:00","start_ts":449348912256647169,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348912256647169,"checkpoint_ts":449348912256647169,"checkpoint_time":"2024-04-26 19:14:26.897"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table ddl_sequence.finish_mark not exists for 8-th check, retry later + set +x + tso='449348913670127617 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348913670127617 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:14:33 CST 2024] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.1052110523.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.t not exists for 1-th check, retry later table event_filter.t1 not exists for 5-th check, retry later table processor_delay.t6 not exists for 3-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/split_region + sink_uri='file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-25640?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/split_region ++ date + echo '[Fri Apr 26 19:14:34 CST 2024] <<<<<< START storage consumer in split_region case >>>>>>' [Fri Apr 26 19:14:34 CST 2024] <<<<<< START storage consumer in split_region case >>>>>> + cd /tmp/tidb_cdc_test/split_region + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/split_region + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/split_region/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/split_region/storage_test/ticdc-split-region-test-25640?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table split_region.test1 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 9-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table partition_table.t not exists for 2-th check, retry later table event_filter.t1 exists table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later table processor_delay.t6 not exists for 4-th check, retry later table split_region.test1 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:36 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1415c498-c5a8-4a2e-8fdc-ddad70e9668b {"id":"1415c498-c5a8-4a2e-8fdc-ddad70e9668b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf748f5 1415c498-c5a8-4a2e-8fdc-ddad70e9668b /tidb/cdc/default/default/upstream/7362132537699175724 {"id":7362132537699175724,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1415c498-c5a8-4a2e-8fdc-ddad70e9668b {"id":"1415c498-c5a8-4a2e-8fdc-ddad70e9668b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf748f5 1415c498-c5a8-4a2e-8fdc-ddad70e9668b /tidb/cdc/default/default/upstream/7362132537699175724 {"id":7362132537699175724,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1415c498-c5a8-4a2e-8fdc-ddad70e9668b {"id":"1415c498-c5a8-4a2e-8fdc-ddad70e9668b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf748f5 1415c498-c5a8-4a2e-8fdc-ddad70e9668b /tidb/cdc/default/default/upstream/7362132537699175724 {"id":7362132537699175724,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.10571.out cli changefeed create --start-ts=449348913670127617 '--sink-uri=file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-9531?protocol=canal-json&enable-tidb-extension=true' table ddl_sequence.finish_mark not exists for 10-th check, retry later Create changefeed successfully! ID: c91451a0-41fc-417d-a9a4-2e4060bbe047 Info: {"upstream_id":7362132537699175724,"namespace":"default","id":"c91451a0-41fc-417d-a9a4-2e4060bbe047","sink_uri":"file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-9531?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:14:37.242740444+08:00","start_ts":449348913670127617,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348913670127617,"checkpoint_ts":449348913670127617,"checkpoint_time":"2024-04-26 19:14:32.289"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t exists table partition_table.t1 exists table partition_table.finish_mark not exists for 1-th check, retry later table processor_delay.t6 exists table processor_delay.t7 not exists for 1-th check, retry later table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/partition_table + sink_uri='file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-9531?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/partition_table ++ date + echo '[Fri Apr 26 19:14:38 CST 2024] <<<<<< START storage consumer in partition_table case >>>>>>' [Fri Apr 26 19:14:38 CST 2024] <<<<<< START storage consumer in partition_table case >>>>>> + cd /tmp/tidb_cdc_test/partition_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/partition_table/storage_test/ticdc-partition-table-test-9531?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table split_region.test1 not exists for 3-th check, retry later table ddl_sequence.finish_mark not exists for 11-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 2-th check, retry later table processor_delay.t7 exists table processor_delay.t8 not exists for 1-th check, retry later table event_filter.t_alter not exists for 2-th check, retry later table split_region.test1 not exists for 4-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 19.47 secs (172796932 bytes/sec) [Pipeline] { [Pipeline] cache table ddl_sequence.finish_mark not exists for 12-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t8 not exists for 2-th check, retry later table event_filter.t_alter not exists for 3-th check, retry later table partition_table.finish_mark not exists for 3-th check, retry later table split_region.test1 not exists for 5-th check, retry later table ddl_sequence.finish_mark not exists for 13-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 4-th check, retry later table processor_delay.t8 not exists for 3-th check, retry later table event_filter.t_alter not exists for 4-th check, retry later table split_region.test1 not exists for 6-th check, retry later table ddl_sequence.finish_mark not exists for 14-th check, retry later table partition_table.finish_mark not exists for 5-th check, retry later table event_filter.t_alter exists VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6875809c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:10303, start at 2024-04-26 19:14:43.908537895 +0800 CST m=+6.319059407 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:43.922 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:43.929 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:43.929 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6875809c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:10303, start at 2024-04-26 19:14:43.908537895 +0800 CST m=+6.319059407 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:43.922 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:43.929 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:43.929 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687583c0000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-rl110, pid:10368, start at 2024-04-26 19:14:44.090744138 +0800 CST m=+6.443336231 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:44.097 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:44.080 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:44.080 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/csv_storage_partition_table/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table split_region.test1 exists table split_region.test2 exists check diff successfully table processor_delay.t8 not exists for 4-th check, retry later table ddl_sequence.finish_mark not exists for 15-th check, retry later table event_filter.finish_mark exists check diff failed 1-th time, retry later table partition_table.finish_mark not exists for 6-th check, retry later table processor_delay.t8 exists table processor_delay.t9 exists table processor_delay.t10 not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 16-th check, retry later check diff failed 2-th time, retry later table partition_table.finish_mark not exists for 7-th check, retry later table processor_delay.t10 not exists for 2-th check, retry later [Fri Apr 26 19:14:50 CST 2024] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1163911641.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc0.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data0 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_sequence.finish_mark not exists for 17-th check, retry later check diff failed 3-th time, retry later table partition_table.finish_mark not exists for 8-th check, retry later table processor_delay.t10 not exists for 3-th check, retry later table ddl_sequence.finish_mark not exists for 18-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:54 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:14:54 CST 2024] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1169111693.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later table partition_table.finish_mark not exists for 9-th check, retry later table processor_delay.t10 not exists for 4-th check, retry later table ddl_sequence.finish_mark not exists for 19-th check, retry later check diff failed 5-th time, retry later table partition_table.finish_mark not exists for 10-th check, retry later [2024/04/26 19:14:55.720 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/04/26 19:14:55.720 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7362131966098464248] [2024/04/26 19:14:55.720 +08:00] [INFO] [client.go:606] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/04/26 19:14:55.720 +08:00] [INFO] [tso_client.go:236] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2024/04/26 19:14:55.721 +08:00] [INFO] [tso_dispatcher.go:359] ["[tso] tso dispatcher created"] [dc-location=global] [2024/04/26 19:14:55.721 +08:00] [INFO] [client.go:612] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/04/26 19:14:55.722 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/04/26 19:14:55.722 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7362131966098464248] [2024/04/26 19:14:55.722 +08:00] [INFO] [client.go:606] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/04/26 19:14:55.722 +08:00] [INFO] [tso_client.go:236] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2024/04/26 19:14:55.723 +08:00] [INFO] [tso_dispatcher.go:359] ["[tso] tso dispatcher created"] [dc-location=global] [2024/04/26 19:14:55.723 +08:00] [INFO] [client.go:612] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/04/26 19:14:55.723 +08:00] [INFO] [tikv_driver.go:197] ["using API V1."] [2024/04/26 19:14:55.724 +08:00] [INFO] [main.go:180] ["genLock started"] [2024/04/26 19:14:55.726 +08:00] [INFO] [store_cache.go:510] ["change store resolve state"] [store=7] [addr=127.0.0.1:20160] [from=unresolved] [to=resolved] [liveness-state=reachable] table partition_table.t exists table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc) 3365112832 bytes in 12.89 secs (261036131 bytes/sec) [Pipeline] { [Pipeline] sh table partition_table.t2 not exists for 2-th check, retry later [Pipeline] sh table processor_delay.t10 not exists for 5-th check, retry later table ddl_sequence.finish_mark not exists for 20-th check, retry later [Pipeline] sh + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:57 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e {"id":"66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a71f 66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e {"id":"66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a71f 66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e {"id":"66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a71f 66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:14:57 CST 2024] <<<<<< START cdc server in csv_storage_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.1173511737.out server --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G00 Run cases: bdr_mode capture_suicide_while_balance_table syncpoint hang_sink_suicide server_config_compatibility kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro lossy_ddl storage_csv_update PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=f989825d-9c0d-4d7a-8722-74a7cdcc9383 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G00 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-f22n4 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-f22n4 xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:14:58 CST 2024] <<<<<< run test case bdr_mode success! >>>>>> [Pipeline] sh check diff failed 6-th time, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G09 Run cases: gc_safepoint changefeed_pause_resume cli savepoint synced_status PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=0beff8c3-1e69-4f06-914e-74b94dc00dad BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G09 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-bj25g GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5 xtiflow_release-7_5_pull_cdc_integration_storage_test_285-bj25g GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G14 Run cases: big_txn changefeed_finish force_replicate_table PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=9fd42bf5-c8ed-4e8e-aff9-b6ad8d020a36 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G14 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-32ksl GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75 xtiflow_release-7_5_pull_cdc_integration_storage_test_285-32ksl GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/big_txn/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G05 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G13 Run cases: tiflash region_merge common_1 PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=113f9677-a70f-497b-aad3-d3824fbca9c8 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 table partition_table.finish_mark not exists for 11-th check, retry later PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G13 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-x64xg GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-x64xg xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table processor_delay.t10 exists table processor_delay.t11 exists table processor_delay.t12 not exists for 1-th check, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G08 Run cases: processor_err_chan changefeed_reconstruct multi_capture PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=60f23b1e-242d-40f7-b4b3-b946d6f6fa7c BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G08 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-609qt GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80 xtiflow_release-7_5_pull_cdc_integration_storage_test_285-609qt GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Run cases: charset_gbk ddl_manager multi_source PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=af37a845-20dd-402e-8ee6-7d796de4ed57 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G05 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-spfqd GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-spfqd xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_sequence.finish_mark not exists for 21-th check, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh storage G17 Run cases: clustered_index processor_resolved_ts_fallback PROW_JOB_ID=b57b8bb6-bdb6-4f9b-832b-13406cde84e1 JENKINS_NODE_COOKIE=dcab3ee5-01ce-45ee-8c2d-e9185aa41c4e BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/ GOLANG_VERSION=1.21.6 HOSTNAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test","buildid":"1783812214617018372","prowjobid":"b57b8bb6-bdb6-4f9b-832b-13406cde84e1","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#285 TEST_GROUP=G17 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214617018372 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_storage_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/285/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=xtiflow_release-7_5_pull_cdc_integration_storage_test_285-sl7qz GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-a17f14038c0f624ee4896b380f0ea8875a3b7d57bef8562b6d5dca2759a31fc3 NODE_LABELS=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01 xtiflow_release-7_5_pull_cdc_integration_storage_test_285-sl7qz GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_storage_test/display/redirect BUILD_NUMBER=285 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:14:59 CST 2024] <<<<<< skip test case clustered_index for storage! >>>>>> find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table partition_table.t2 not exists for 3-th check, retry later check diff failed 7-th time, retry later table partition_table.finish_mark not exists for 12-th check, retry later table processor_delay.t12 not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:00 GMT < Content-Length: 1144 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e {"id":"66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/ce113f25-5eda-4c33-9635-650efa610eea {"id":"ce113f25-5eda-4c33-9635-650efa610eea","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a71f 66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a733 ce113f25-5eda-4c33-9635-650efa610eea /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e {"id":"66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/ce113f25-5eda-4c33-9635-650efa610eea {"id":"ce113f25-5eda-4c33-9635-650efa610eea","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a71f 66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a733 ce113f25-5eda-4c33-9635-650efa610eea /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e {"id":"66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6900937f-e7c0-4fda-bfa6-dc8cce960325 {"id":"6900937f-e7c0-4fda-bfa6-dc8cce960325","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/ce113f25-5eda-4c33-9635-650efa610eea {"id":"ce113f25-5eda-4c33-9635-650efa610eea","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a6f6 6900937f-e7c0-4fda-bfa6-dc8cce960325 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a71f 66fc054a-4a05-4944-8dd4-bbe6c9e4eb5e /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d31a733 ce113f25-5eda-4c33-9635-650efa610eea /tidb/cdc/default/default/upstream/7362132609482213622 {"id":7362132609482213622,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.csv_storage_partition_table.cli.11782.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml Create changefeed successfully! ID: 0977cf9c-9404-404b-b88d-be53db555400 Info: {"upstream_id":7362132609482213622,"namespace":"default","id":"0977cf9c-9404-404b-b88d-be53db555400","sink_uri":"file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:15:00.912297749+08:00","start_ts":449348921138348048,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348921138348048,"checkpoint_ts":449348921138348048,"checkpoint_time":"2024-04-26 19:15:00.778"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... table partition_table.t2 not exists for 4-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:01 CST 2024] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table ddl_sequence.finish_mark not exists for 22-th check, retry later check diff failed 1-th time, retry later check diff failed 2-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/big_txn Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... + set +x start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check diff failed 8-th time, retry later table partition_table.finish_mark not exists for 13-th check, retry later table processor_delay.t12 not exists for 3-th check, retry later table partition_table.t2 not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table ddl_sequence.finish_mark not exists for 23-th check, retry later check diff failed 3-th time, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: storage... <<================= kafka downstream isn't support syncpoint record [Fri Apr 26 19:15:03 CST 2024] <<<<<< run test case syncpoint success! >>>>>> Verifying downstream PD is started... table partition_table.finish_mark not exists for 14-th check, retry later check diff failed 9-th time, retry later table processor_delay.t12 not exists for 4-th check, retry later Verifying downstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table partition_table.t2 not exists for 6-th check, retry later Verifying downstream PD is started... table ddl_sequence.finish_mark not exists for 24-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table partition_table.finish_mark not exists for 15-th check, retry later [2024/04/26 19:15:05.742 +08:00] [INFO] [main.go:196] ["genLock done"] [2024/04/26 19:15:05.742 +08:00] [INFO] [pd_service_discovery.go:550] ["[pd] exit member loop due to context canceled"] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_dispatcher.go:214] ["exit tso requests cancel loop"] [2024/04/26 19:15:05.742 +08:00] [INFO] [resource_manager_client.go:295] ["[resource manager] exit resource token dispatcher"] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_dispatcher.go:268] ["exit tso dispatcher loop"] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_client.go:140] ["closing tso client"] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_dispatcher.go:455] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_dispatcher.go:380] ["[tso] exit tso dispatcher"] [dc-location=global] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_batch_controller.go:158] ["[pd] clear the tso batch controller"] [max-batch-size=10000] [best-batch-size=1] [collected-request-count=0] [pending-request-count=0] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_client.go:145] ["close tso client"] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_batch_controller.go:158] ["[pd] clear the tso batch controller"] [max-batch-size=10000] [best-batch-size=1] [collected-request-count=0] [pending-request-count=0] [2024/04/26 19:15:05.742 +08:00] [INFO] [tso_client.go:155] ["tso client is closed"] [2024/04/26 19:15:05.742 +08:00] [INFO] [pd_service_discovery.go:637] ["[pd] close pd service discovery client"] wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 not exists for 7-th check, retry later wait process cdc.test exit for 3-th time... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff successfully wait process cdc.test exit for 1-th time... table processor_delay.t12 not exists for 5-th check, retry later table ddl_sequence.finish_mark not exists for 25-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/hang_sink_suicide/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:07 CST 2024] <<<<<< run test case hang_sink_suicide success! >>>>>> Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:15:07 CST 2024] <<<<<< run test case split_region success! >>>>>> wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:08 CST 2024] <<<<<< run test case event_filter success! >>>>>> Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 16-th check, retry later table processor_delay.t12 exists table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later table ddl_sequence.finish_mark exists Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table partition_table.t2 not exists for 8-th check, retry later check diff successfully Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:10 CST 2024] <<<<<< run test case server_config_compatibility success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:10 CST 2024] <<<<<< run test case ddl_sequence success! >>>>>> table partition_table.finish_mark exists table processor_delay.t14 not exists for 2-th check, retry later table partition_table.t2 not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t14 not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687733600003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:1225, start at 2024-04-26 19:15:11.70703092 +0800 CST m=+5.162115314 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:11.714 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:11.704 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:11.704 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687733600003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:1225, start at 2024-04-26 19:15:11.70703092 +0800 CST m=+5.162115314 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:11.714 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:11.704 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:11.704 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687734d00015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:1291, start at 2024-04-26 19:15:11.843911137 +0800 CST m=+5.212244119 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:11.850 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:11.845 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:11.845 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:13 CST 2024] <<<<<< run test case kafka_big_messages success! >>>>>> table partition_table.t2 not exists for 10-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... table test.t2 not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:15:14 CST 2024] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t14 not exists for 4-th check, retry later [Fri Apr 26 19:15:14 CST 2024] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.25232525.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 not exists for 11-th check, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687746cc000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:1230, start at 2024-04-26 19:15:12.969396634 +0800 CST m=+5.374091939 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:12.976 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:12.947 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:12.947 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687746cc000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:1230, start at 2024-04-26 19:15:12.969396634 +0800 CST m=+5.374091939 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:12.976 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:12.947 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:12.947 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687746c40015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:1308, start at 2024-04-26 19:15:12.992788037 +0800 CST m=+5.347874377 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:12.999 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:12.996 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:12.996 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/big_txn/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/big_txn/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68774d900013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36, pid:1120, start at 2024-04-26 19:15:13.425495278 +0800 CST m=+5.404742013 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:13.433 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:13.430 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:13.430 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68774d900013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36, pid:1120, start at 2024-04-26 19:15:13.425495278 +0800 CST m=+5.404742013 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:13.433 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:13.430 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:13.430 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68774efc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36, pid:1164, start at 2024-04-26 19:15:13.519286298 +0800 CST m=+5.426053675 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:13.526 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:13.521 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:13.521 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:15 CST 2024] <<<<<< run test case kafka_compression success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68777ba80001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz, pid:1136, start at 2024-04-26 19:15:16.33427033 +0800 CST m=+7.933840217 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:16.354 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:16.330 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:16.330 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.t2 not exists for 12-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687754e00019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01, pid:1181, start at 2024-04-26 19:15:13.891912342 +0800 CST m=+5.258858070 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:13.899 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:13.897 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:13.897 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687754e00019 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01, pid:1181, start at 2024-04-26 19:15:13.891912342 +0800 CST m=+5.258858070 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:13.899 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:13.897 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:13.897 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687756680014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-2hg01, pid:1248, start at 2024-04-26 19:15:13.981684511 +0800 CST m=+5.296412980 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:13.990 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:13.946 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:13.946 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t14 not exists for 5-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68778b240002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:1139, start at 2024-04-26 19:15:17.3230524 +0800 CST m=+7.902401758 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:17.340 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:17.321 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:17.321 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.t2 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:17 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70d5dfef-9acc-4418-8a5e-8299fdcad55f {"id":"70d5dfef-9acc-4418-8a5e-8299fdcad55f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da4c6f0 70d5dfef-9acc-4418-8a5e-8299fdcad55f /tidb/cdc/default/default/upstream/7362132737071396878 {"id":7362132737071396878,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70d5dfef-9acc-4418-8a5e-8299fdcad55f {"id":"70d5dfef-9acc-4418-8a5e-8299fdcad55f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da4c6f0 70d5dfef-9acc-4418-8a5e-8299fdcad55f /tidb/cdc/default/default/upstream/7362132737071396878 {"id":7362132737071396878,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/70d5dfef-9acc-4418-8a5e-8299fdcad55f {"id":"70d5dfef-9acc-4418-8a5e-8299fdcad55f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da4c6f0 70d5dfef-9acc-4418-8a5e-8299fdcad55f /tidb/cdc/default/default/upstream/7362132737071396878 {"id":7362132737071396878,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/gc_safepoint + sink_uri='file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-31413?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/gc_safepoint ++ date + echo '[Fri Apr 26 19:15:18 CST 2024] <<<<<< START storage consumer in gc_safepoint case >>>>>>' [Fri Apr 26 19:15:18 CST 2024] <<<<<< START storage consumer in gc_safepoint case >>>>>> + cd /tmp/tidb_cdc_test/gc_safepoint + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/gc_safepoint + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-31413?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' 0 [Fri Apr 26 19:15:17 CST 2024] <<<<<< START cdc server in big_txn case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.25542556.out server --log-file /tmp/tidb_cdc_test/big_txn/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/big_txn/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2452.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68777ba80001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz, pid:1136, start at 2024-04-26 19:15:16.33427033 +0800 CST m=+7.933840217 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:16.354 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:16.330 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:16.330 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68777d100007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz, pid:1169, start at 2024-04-26 19:15:16.43754876 +0800 CST m=+7.843453813 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:16.459 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:16.470 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:16.470 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:18 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.25652567.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 26 19:15:19 CST 2024] <<<<<< run test case kafka_messages success! >>>>>> TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t14 exists table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later table partition_table.t2 not exists for 13-th check, retry later check diff failed 1-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68778b240002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:1139, start at 2024-04-26 19:15:17.3230524 +0800 CST m=+7.902401758 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:17.340 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:17.321 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:17.321 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687787dc000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:1171, start at 2024-04-26 19:15:17.152765947 +0800 CST m=+7.543028210 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:17.169 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:17.161 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:17.161 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... table test.t2 not exists for 5-th check, retry later + set +x + tso='449348925831512065 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348925831512065 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:15:20 CST 2024] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.24792481.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m table processor_delay.t16 not exists for 2-th check, retry later table partition_table.t2 exists table partition_table.finish_mark not exists for 1-th check, retry later check diff failed 2-th time, retry later [Pipeline] // cache + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:21 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/64538ce1-56f8-4191-a56e-81be1bb743db {"id":"64538ce1-56f8-4191-a56e-81be1bb743db","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da6e6f2 64538ce1-56f8-4191-a56e-81be1bb743db /tidb/cdc/default/default/upstream/7362132740395305762 {"id":7362132740395305762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/64538ce1-56f8-4191-a56e-81be1bb743db {"id":"64538ce1-56f8-4191-a56e-81be1bb743db","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da6e6f2 64538ce1-56f8-4191-a56e-81be1bb743db /tidb/cdc/default/default/upstream/7362132740395305762 {"id":7362132740395305762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/64538ce1-56f8-4191-a56e-81be1bb743db {"id":"64538ce1-56f8-4191-a56e-81be1bb743db","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da6e6f2 64538ce1-56f8-4191-a56e-81be1bb743db /tidb/cdc/default/default/upstream/7362132740395305762 {"id":7362132740395305762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:15:21 CST 2024] <<<<<< START cdc server in big_txn case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.26022604.out server --log-file /tmp/tidb_cdc_test/big_txn/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/big_txn/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:21 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/64538ce1-56f8-4191-a56e-81be1bb743db {"id":"64538ce1-56f8-4191-a56e-81be1bb743db","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da6e6f2 64538ce1-56f8-4191-a56e-81be1bb743db /tidb/cdc/default/default/upstream/7362132740395305762 {"id":7362132740395305762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/64538ce1-56f8-4191-a56e-81be1bb743db {"id":"64538ce1-56f8-4191-a56e-81be1bb743db","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da6e6f2 64538ce1-56f8-4191-a56e-81be1bb743db /tidb/cdc/default/default/upstream/7362132740395305762 {"id":7362132740395305762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/64538ce1-56f8-4191-a56e-81be1bb743db {"id":"64538ce1-56f8-4191-a56e-81be1bb743db","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da6e6f2 64538ce1-56f8-4191-a56e-81be1bb743db /tidb/cdc/default/default/upstream/7362132740395305762 {"id":7362132740395305762,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.cli.2624.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/big_txn/storage_test/ticdc-big-txn-test-21441?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 4c6e8777-22aa-4740-892f-4352ad084887 Info: {"upstream_id":7362132740395305762,"namespace":"default","id":"4c6e8777-22aa-4740-892f-4352ad084887","sink_uri":"file:///tmp/tidb_cdc_test/big_txn/storage_test/ticdc-big-txn-test-21441?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:15:21.506567303+08:00","start_ts":449348926543757320,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348926543757320,"checkpoint_ts":449348926543757320,"checkpoint_time":"2024-04-26 19:15:21.398"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] // cache [Fri Apr 26 19:15:21 CST 2024] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.24082410.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.t2 not exists for 6-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:22 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f73e442-3181-4318-8507-3530c1e0fc49 {"id":"3f73e442-3181-4318-8507-3530c1e0fc49","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8daf3 3f73e442-3181-4318-8507-3530c1e0fc49 /tidb/cdc/default/default/upstream/7362132735425089381 {"id":7362132735425089381,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f73e442-3181-4318-8507-3530c1e0fc49 {"id":"3f73e442-3181-4318-8507-3530c1e0fc49","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8daf3 3f73e442-3181-4318-8507-3530c1e0fc49 /tidb/cdc/default/default/upstream/7362132735425089381 {"id":7362132735425089381,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f73e442-3181-4318-8507-3530c1e0fc49 {"id":"3f73e442-3181-4318-8507-3530c1e0fc49","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8daf3 3f73e442-3181-4318-8507-3530c1e0fc49 /tidb/cdc/default/default/upstream/7362132735425089381 {"id":7362132735425089381,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.2612.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:22 CST 2024] <<<<<< run test case kafka_sink_error_resume success! >>>>>> Create changefeed successfully! ID: a9b12899-850e-48d5-b30e-0222c1d76d47 Info: {"upstream_id":7362132735425089381,"namespace":"default","id":"a9b12899-850e-48d5-b30e-0222c1d76d47","sink_uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:15:22.453178529+08:00","start_ts":449348926792531974,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348926792531974,"checkpoint_ts":449348926792531974,"checkpoint_time":"2024-04-26 19:15:22.347"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] } [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) + set +x table processor_delay.t16 not exists for 3-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] // dir [Pipeline] // dir [Pipeline] // cache table partition_table.finish_mark not exists for 2-th check, retry later [Pipeline] // cache + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:23 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5048b4e5-1155-4284-bdcf-b4e6ebd2727b {"id":"5048b4e5-1155-4284-bdcf-b4e6ebd2727b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da897f6 5048b4e5-1155-4284-bdcf-b4e6ebd2727b /tidb/cdc/default/default/upstream/7362132741919613013 {"id":7362132741919613013,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5048b4e5-1155-4284-bdcf-b4e6ebd2727b {"id":"5048b4e5-1155-4284-bdcf-b4e6ebd2727b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da897f6 5048b4e5-1155-4284-bdcf-b4e6ebd2727b /tidb/cdc/default/default/upstream/7362132741919613013 {"id":7362132741919613013,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5048b4e5-1155-4284-bdcf-b4e6ebd2727b {"id":"5048b4e5-1155-4284-bdcf-b4e6ebd2727b","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da897f6 5048b4e5-1155-4284-bdcf-b4e6ebd2727b /tidb/cdc/default/default/upstream/7362132741919613013 {"id":7362132741919613013,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2525.out cli changefeed create --start-ts=449348925831512065 '--sink-uri=file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-14440?protocol=canal-json&enable-tidb-extension=true' [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // withCredentials Create changefeed successfully! ID: 5b3a7b61-49ad-4470-804f-ed764cd28b27 Info: {"upstream_id":7362132741919613013,"namespace":"default","id":"5b3a7b61-49ad-4470-804f-ed764cd28b27","sink_uri":"file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-14440?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:15:23.576277636+08:00","start_ts":449348925831512065,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348925831512065,"checkpoint_ts":449348925831512065,"checkpoint_time":"2024-04-26 19:15:18.681"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] // withCredentials [Pipeline] // dir [Pipeline] // dir [Pipeline] } table test.t2 not exists for 7-th check, retry later [Pipeline] } [Pipeline] } [Pipeline] } + set +x + workdir=/tmp/tidb_cdc_test/processor_resolved_ts_fallback + sink_uri='file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_resolved_ts_fallback ++ date + echo '[Fri Apr 26 19:15:23 CST 2024] <<<<<< START storage consumer in processor_resolved_ts_fallback case >>>>>>' [Fri Apr 26 19:15:23 CST 2024] <<<<<< START storage consumer in processor_resolved_ts_fallback case >>>>>> + cd /tmp/tidb_cdc_test/processor_resolved_ts_fallback + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_resolved_ts_fallback + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Pipeline] // timeout check diff failed 3-th time, retry later [Pipeline] // timeout [Pipeline] // withCredentials [Pipeline] // withCredentials [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // stage [Pipeline] // stage + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:24 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98c750f6-9e18-475e-9bb7-019dbb6e8b74 {"id":"98c750f6-9e18-475e-9bb7-019dbb6e8b74","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dab7af6 98c750f6-9e18-475e-9bb7-019dbb6e8b74 /tidb/cdc/default/default/upstream/7362132738156545685 {"id":7362132738156545685,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98c750f6-9e18-475e-9bb7-019dbb6e8b74 {"id":"98c750f6-9e18-475e-9bb7-019dbb6e8b74","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dab7af6 98c750f6-9e18-475e-9bb7-019dbb6e8b74 /tidb/cdc/default/default/upstream/7362132738156545685 {"id":7362132738156545685,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/98c750f6-9e18-475e-9bb7-019dbb6e8b74 {"id":"98c750f6-9e18-475e-9bb7-019dbb6e8b74","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dab7af6 98c750f6-9e18-475e-9bb7-019dbb6e8b74 /tidb/cdc/default/default/upstream/7362132738156545685 {"id":7362132738156545685,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "dotransactions"="false" "mysql.port"="4000" "readallfields"="true" "workload"="core" "mysql.host"="127.0.0.1" "recordcount"="5000" "scanproportion"="0" "fieldcount"="100" "operationcount"="0" "requestdistribution"="uniform" "threadcount"="1" "mysql.user"="root" "mysql.db"="big_txn" "insertproportion"="0" "updateproportion"="0" "readproportion"="0" ********************************************** [Pipeline] // timeout [Pipeline] // timeout [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // container Create changefeed successfully! ID: 451e748e-b795-4813-b75d-242bddf5d7b2 Info: {"upstream_id":7362132738156545685,"namespace":"default","id":"451e748e-b795-4813-b75d-242bddf5d7b2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:15:24.687422014+08:00","start_ts":449348926512562177,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348926512562177,"checkpoint_ts":449348926512562177,"checkpoint_time":"2024-04-26 19:15:21.279"} [Pipeline] // container [Pipeline] // stage [Pipeline] // stage [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } + workdir=/tmp/tidb_cdc_test/charset_gbk + sink_uri=mysql://normal:123456@127.0.0.1:3306/ + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/charset_gbk ++ date + echo '[Fri Apr 26 19:15:24 CST 2024] <<<<<< START storage consumer in charset_gbk case >>>>>>' [Fri Apr 26 19:15:24 CST 2024] <<<<<< START storage consumer in charset_gbk case >>>>>> + cd /tmp/tidb_cdc_test/charset_gbk + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/charset_gbk + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/charset_gbk/cdc_storage_consumer.log --log-level debug --upstream-uri mysql://normal:123456@127.0.0.1:3306/ --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Pipeline] // withEnv [Pipeline] // withEnv + set +x + workdir=/tmp/tidb_cdc_test/tiflash + sink_uri='file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-14440?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/tiflash ++ date + echo '[Fri Apr 26 19:15:24 CST 2024] <<<<<< START storage consumer in tiflash case >>>>>>' [Fri Apr 26 19:15:24 CST 2024] <<<<<< START storage consumer in tiflash case >>>>>> + cd /tmp/tidb_cdc_test/tiflash + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/tiflash + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/tiflash/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/tiflash/storage_test/ticdc-tiflash-test-14440?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Pipeline] // container [Pipeline] // container table partition_table.finish_mark not exists for 3-th check, retry later [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // node start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table processor_delay.t16 not exists for 4-th check, retry later find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:25 CST 2024] <<<<<< run test case mq_sink_lost_callback success! >>>>>> table test.t2 not exists for 8-th check, retry later [Pipeline] // node check diff failed 4-th time, retry later [Pipeline] // withEnv cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table processor_resolved_ts_fallback.t1 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: storage... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Fri Apr 26 19:15:25 CST 2024] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/sourcemanager/puller/ProcessorAddTableError=1*return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.25102512.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table cdc_tiflash_test.multi_data_type not exists for 1-th check, retry later [Pipeline] // podTemplate [Pipeline] // podTemplate [Pipeline] // node [Pipeline] // node [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] // withEnv [Pipeline] // withEnv [Pipeline] // podTemplate [Pipeline] // podTemplate [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } table partition_table.finish_mark not exists for 4-th check, retry later [Pipeline] // stage check diff failed 5-th time, retry later [Pipeline] // stage [Pipeline] // withEnv You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend [Pipeline] // withEnv [Pipeline] } [Pipeline] } [Pipeline] } [Pipeline] } Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide table processor_delay.t16 not exists for 5-th check, retry later [Pipeline] // stage [Pipeline] // stage find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:28 CST 2024] <<<<<< run test case mq_sink_dispatcher success! >>>>>> table test.t2 not exists for 9-th check, retry later [Pipeline] } [Pipeline] } * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17C9D04524812BCC < X-Xss-Protection: 1; mode=block < Date: Fri, 26 Apr 2024 11:15:28 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created [Fri Apr 26 19:15:28 CST 2024] <<<<<< run test case multi_tables_ddl success! >>>>>> Exiting on signal: INTERRUPT + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > table cdc_tiflash_test.multi_data_type not exists for 2-th check, retry later table processor_resolved_ts_fallback.t1 not exists for 2-th check, retry later < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:28 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c066e71-9cd7-473f-aef7-b1856a618804 {"id":"2c066e71-9cd7-473f-aef7-b1856a618804","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dab766c 2c066e71-9cd7-473f-aef7-b1856a618804 /tidb/cdc/default/default/upstream/7362132736675945007 {"id":7362132736675945007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c066e71-9cd7-473f-aef7-b1856a618804 {"id":"2c066e71-9cd7-473f-aef7-b1856a618804","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dab766c 2c066e71-9cd7-473f-aef7-b1856a618804 /tidb/cdc/default/default/upstream/7362132736675945007 {"id":7362132736675945007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2c066e71-9cd7-473f-aef7-b1856a618804 {"id":"2c066e71-9cd7-473f-aef7-b1856a618804","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dab766c 2c066e71-9cd7-473f-aef7-b1856a618804 /tidb/cdc/default/default/upstream/7362132736675945007 {"id":7362132736675945007,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/processor_err_chan + sink_uri='file:///tmp/tidb_cdc_test/processor_err_chan/storage_test/ticdc-processor-err-chan-13654?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/processor_err_chan ++ date + echo '[Fri Apr 26 19:15:29 CST 2024] <<<<<< START storage consumer in processor_err_chan case >>>>>>' [Fri Apr 26 19:15:29 CST 2024] <<<<<< START storage consumer in processor_err_chan case >>>>>> + cd /tmp/tidb_cdc_test/processor_err_chan + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/processor_err_chan + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/processor_err_chan/storage_test/ticdc-processor-err-chan-13654?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 41551dde-6331-405e-a214-af1d05908bf4 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=41551dde-6331-405e-a214-af1d05908bf4 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 41551dde-6331-405e-a214-af1d05908bf4 -s + info='{ "upstream_id": 7362132736675945007, "namespace": "default", "id": "41551dde-6331-405e-a214-af1d05908bf4", "state": "normal", "checkpoint_tso": 449348928515604482, "checkpoint_time": "2024-04-26 19:15:28.920", "error": null }' + echo '{ "upstream_id": 7362132736675945007, "namespace": "default", "id": "41551dde-6331-405e-a214-af1d05908bf4", "state": "normal", "checkpoint_tso": 449348928515604482, "checkpoint_time": "2024-04-26 19:15:28.920", "error": null }' { "upstream_id": 7362132736675945007, "namespace": "default", "id": "41551dde-6331-405e-a214-af1d05908bf4", "state": "normal", "checkpoint_tso": 449348928515604482, "checkpoint_time": "2024-04-26 19:15:28.920", "error": null } ++ echo '{' '"upstream_id":' 7362132736675945007, '"namespace":' '"default",' '"id":' '"41551dde-6331-405e-a214-af1d05908bf4",' '"state":' '"normal",' '"checkpoint_tso":' 449348928515604482, '"checkpoint_time":' '"2024-04-26' '19:15:28.920",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7362132736675945007, '"namespace":' '"default",' '"id":' '"41551dde-6331-405e-a214-af1d05908bf4",' '"state":' '"normal",' '"checkpoint_tso":' 449348928515604482, '"checkpoint_time":' '"2024-04-26' '19:15:28.920",' '"error":' null '}' + message=null + [[ ! null =~ null ]] run task successfully table partition_table.finish_mark not exists for 5-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table processor_delay.t16 exists table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later table test.t2 not exists for 10-th check, retry later check diff successfully check_safepoint_forward http://127.0.0.1:2379 7362132737071396878 449348928445874179 449348925667409925 check diff failed 1-th time, retry later table cdc_tiflash_test.multi_data_type not exists for 3-th check, retry later table processor_resolved_ts_fallback.t1 not exists for 3-th check, retry later run task successfully check_changefeed_state http://127.0.0.1:2379 650f36b9-3ed3-49a0-8a2e-47842d39de68 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=650f36b9-3ed3-49a0-8a2e-47842d39de68 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 650f36b9-3ed3-49a0-8a2e-47842d39de68 -s + info='{ "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "stopped", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null }' + echo '{ "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "stopped", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null }' { "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "stopped", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null } ++ echo '{' '"upstream_id":' 7362132737071396878, '"namespace":' '"default",' '"id":' '"650f36b9-3ed3-49a0-8a2e-47842d39de68",' '"state":' '"stopped",' '"checkpoint_tso":' 449348928970424324, '"checkpoint_time":' '"2024-04-26' '19:15:30.655",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7362132737071396878, '"namespace":' '"default",' '"id":' '"650f36b9-3ed3-49a0-8a2e-47842d39de68",' '"state":' '"stopped",' '"checkpoint_tso":' 449348928970424324, '"checkpoint_time":' '"2024-04-26' '19:15:30.655",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7362132737071396878 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:31 CST 2024] <<<<<< run test case kafka_column_selector success! >>>>>> table partition_table.finish_mark not exists for 6-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t18 not exists for 2-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } table test.t2 not exists for 11-th check, retry later Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials safepoint is unexpected forward: 449348928708018179 449348925667409925 -> 449348928970424323 449348925667409925 run task failed 1-th time, retry later [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } table cdc_tiflash_test.multi_data_type not exists for 4-th check, retry later [Pipeline] // withEnv table processor_resolved_ts_fallback.t1 not exists for 4-th check, retry later [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } check diff failed 2-th time, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table partition_table.finish_mark not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t18 not exists for 3-th check, retry later table test.t2 not exists for 12-th check, retry later INSERT - Takes(s): 10.0, Count: 2395, OPS: 240.0, Avg(us): 3879, Min(us): 1420, Max(us): 82946, 95th(us): 4000, 99th(us): 71000 + workdir=/tmp/tidb_cdc_test/csv_storage_partition_table + sink_uri='file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/csv_storage_partition_table ++ date + echo '[Fri Apr 26 19:15:33 CST 2024] <<<<<< START storage consumer in csv_storage_partition_table case >>>>>>' [Fri Apr 26 19:15:33 CST 2024] <<<<<< START storage consumer in csv_storage_partition_table case >>>>>> + cd /tmp/tidb_cdc_test/csv_storage_partition_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/csv_storage_partition_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/csv_storage_partition_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/csv_storage_partition_table/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/csv_storage_partition_table/conf/changefeed.toml table processor_resolved_ts_fallback.t1 exists [Fri Apr 26 19:15:34 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.27492751.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:15:34 CST 2024] <<<<<< run test case kafka_column_selector_avro success! >>>>>> check_safepoint_equal http://127.0.0.1:2379 7362132737071396878 table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 not exists for 1-th check, retry later table cdc_tiflash_test.multi_data_type not exists for 5-th check, retry later table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later table partition_table.finish_mark not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later table processor_delay.t18 not exists for 4-th check, retry later table test.t2 not exists for 13-th check, retry later table cdc_tiflash_test.multi_data_type exists check diff successfully find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... run task successfully check diff failed 4-th time, retry later wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 2-th check, retry later table partition_table.finish_mark not exists for 9-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:37 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/a9b12899-850e-48d5-b30e-0222c1d76d47 {UpstreamID:7362132735425089381 Namespace:default ID:a9b12899-850e-48d5-b30e-0222c1d76d47 SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:15:22.453178529 +0800 CST StartTs:449348926792531974 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002d6ac60 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348926818746369} {CheckpointTs:449348930554298371 MinTableBarrierTs:449348930554298371 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f73e442-3181-4318-8507-3530c1e0fc49 {"id":"3f73e442-3181-4318-8507-3530c1e0fc49","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/acf76d8c-7ff6-461f-9fb6-b8f01efc997b {"id":"acf76d8c-7ff6-461f-9fb6-b8f01efc997b","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8daf3 3f73e442-3181-4318-8507-3530c1e0fc49 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8db72 acf76d8c-7ff6-461f-9fb6-b8f01efc997b /tidb/cdc/default/default/changefeed/info/a9b12899-850e-48d5-b30e-0222c1d76d47 {"upstream-id":7362132735425089381,"namespace":"default","changefeed-id":"a9b12899-850e-48d5-b30e-0222c1d76d47","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:15:22.453178529+08:00","start-ts":449348926792531974,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348926818746369} /tidb/cdc/default/default/changefeed/status/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":449348930554298371,"min-table-barrier-ts":449348930554298371,"admin-job-type":0} /tidb/cdc/default/default/task/position/3f73e442-3181-4318-8507-3530c1e0fc49/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/acf76d8c-7ff6-461f-9fb6-b8f01efc997b/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132735425089381 {"id":7362132735425089381,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/a9b12899-850e-48d5-b30e-0222c1d76d47 {UpstreamID:7362132735425089381 Namespace:default ID:a9b12899-850e-48d5-b30e-0222c1d76d47 SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:15:22.453178529 +0800 CST StartTs:449348926792531974 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002d6ac60 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348926818746369} {CheckpointTs:449348930554298371 MinTableBarrierTs:449348930554298371 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f73e442-3181-4318-8507-3530c1e0fc49 {"id":"3f73e442-3181-4318-8507-3530c1e0fc49","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/acf76d8c-7ff6-461f-9fb6-b8f01efc997b {"id":"acf76d8c-7ff6-461f-9fb6-b8f01efc997b","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8daf3 3f73e442-3181-4318-8507-3530c1e0fc49 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8db72 acf76d8c-7ff6-461f-9fb6-b8f01efc997b /tidb/cdc/default/default/changefeed/info/a9b12899-850e-48d5-b30e-0222c1d76d47 {"upstream-id":7362132735425089381,"namespace":"default","changefeed-id":"a9b12899-850e-48d5-b30e-0222c1d76d47","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:15:22.453178529+08:00","start-ts":449348926792531974,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348926818746369} /tidb/cdc/default/default/changefeed/status/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":449348930554298371,"min-table-barrier-ts":449348930554298371,"admin-job-type":0} /tidb/cdc/default/default/task/position/3f73e442-3181-4318-8507-3530c1e0fc49/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/acf76d8c-7ff6-461f-9fb6-b8f01efc997b/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132735425089381 {"id":73621+ grep -q 'failed to get info:' 32735425089381,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/a9b12899-850e-48d5-b30e-0222c1d76d47 {UpstreamID:7362132735425089381 Namespace:default ID:a9b12899-850e-48d5-b30e-0222c1d76d47 SinkURI:file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:15:22.453178529 +0800 CST StartTs:449348926792531974 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002d6ac60 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348926818746369} {CheckpointTs:449348930554298371 MinTableBarrierTs:449348930554298371 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3f73e442-3181-4318-8507-3530c1e0fc49 {"id":"3f73e442-3181-4318-8507-3530c1e0fc49","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/acf76d8c-7ff6-461f-9fb6-b8f01efc997b {"id":"acf76d8c-7ff6-461f-9fb6-b8f01efc997b","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8daf3 3f73e442-3181-4318-8507-3530c1e0fc49 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1da8db72 acf76d8c-7ff6-461f-9fb6-b8f01efc997b /tidb/cdc/default/default/changefeed/info/a9b12899-850e-48d5-b30e-0222c1d76d47 {"upstream-id":7362132735425089381,"namespace":"default","changefeed-id":"a9b12899-850e-48d5-b30e-0222c1d76d47","sink-uri":"file:///tmp/tidb_cdc_test/processor_resolved_ts_fallback/storage_test/ticdc-processor-resolved-ts-fallback-test-1297?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:15:22.453178529+08:00","start-ts":449348926792531974,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348926818746369} /tidb/cdc/default/default/changefeed/status/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":449348930554298371,"min-table-barrier-ts":449348930554298371,"admin-job-type":0} /tidb/cdc/default/default/task/position/3f73e442-3181-4318-8507-3530c1e0fc49/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/acf76d8c-7ff6-461f-9fb6-b8f01efc997b/a9b12899-850e-48d5-b30e-0222c1d76d47 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132735425089381 {"id":7362132735425089381,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 650f36b9-3ed3-49a0-8a2e-47842d39de68 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=650f36b9-3ed3-49a0-8a2e-47842d39de68 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 650f36b9-3ed3-49a0-8a2e-47842d39de68 -s + info='{ "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "normal", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null }' + echo '{ "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "normal", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null }' { "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "normal", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null } ++ echo '{' '"upstream_id":' 7362132737071396878, '"namespace":' '"default",' '"id":' '"650f36b9-3ed3-49a0-8a2e-47842d39de68",' '"state":' '"normal",' '"checkpoint_tso":' 449348928970424324, '"checkpoint_time":' '"2024-04-26' '19:15:30.655",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132737071396878, '"namespace":' '"default",' '"id":' '"650f36b9-3ed3-49a0-8a2e-47842d39de68",' '"state":' '"normal",' '"checkpoint_tso":' 449348928970424324, '"checkpoint_time":' '"2024-04-26' '19:15:30.655",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7362132737071396878 449348928970424323 449348928970424324 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6878ba8c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:9863, start at 2024-04-26 19:15:36.762392191 +0800 CST m=+5.371520187 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:36.771 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:36.739 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:36.739 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table processor_delay.t18 not exists for 5-th check, retry later table test.t2 exists check diff successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:38 CST 2024] <<<<<< run test case tiflash success! >>>>>> wait process cdc.test exit for 1-th time... table processor_resolved_ts_fallback.t2 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... run task successfully check_changefeed_state http://127.0.0.1:2379 650f36b9-3ed3-49a0-8a2e-47842d39de68 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=650f36b9-3ed3-49a0-8a2e-47842d39de68 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 650f36b9-3ed3-49a0-8a2e-47842d39de68 -s + info='{ "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "stopped", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null }' + echo '{ "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "stopped", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null }' { "upstream_id": 7362132737071396878, "namespace": "default", "id": "650f36b9-3ed3-49a0-8a2e-47842d39de68", "state": "stopped", "checkpoint_tso": 449348928970424324, "checkpoint_time": "2024-04-26 19:15:30.655", "error": null } ++ echo '{' '"upstream_id":' 7362132737071396878, '"namespace":' '"default",' '"id":' '"650f36b9-3ed3-49a0-8a2e-47842d39de68",' '"state":' '"stopped",' '"checkpoint_tso":' 449348928970424324, '"checkpoint_time":' '"2024-04-26' '19:15:30.655",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7362132737071396878, '"namespace":' '"default",' '"id":' '"650f36b9-3ed3-49a0-8a2e-47842d39de68",' '"state":' '"stopped",' '"checkpoint_tso":' 449348928970424324, '"checkpoint_time":' '"2024-04-26' '19:15:30.655",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state http://127.0.0.1:2379 936a2f5c-2c01-465b-a65e-dd3a7830a9ab normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=936a2f5c-2c01-465b-a65e-dd3a7830a9ab + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 936a2f5c-2c01-465b-a65e-dd3a7830a9ab -s cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:39 CST 2024] <<<<<< run test case resolve_lock success! >>>>>> table test.finish_mark not exists for 3-th check, retry later table partition_table.finish_mark not exists for 10-th check, retry later + info='{ "upstream_id": 7362132737071396878, "namespace": "default", "id": "936a2f5c-2c01-465b-a65e-dd3a7830a9ab", "state": "normal", "checkpoint_tso": 449348931263922179, "checkpoint_time": "2024-04-26 19:15:39.404", "error": null }' + echo '{ "upstream_id": 7362132737071396878, "namespace": "default", "id": "936a2f5c-2c01-465b-a65e-dd3a7830a9ab", "state": "normal", "checkpoint_tso": 449348931263922179, "checkpoint_time": "2024-04-26 19:15:39.404", "error": null }' { "upstream_id": 7362132737071396878, "namespace": "default", "id": "936a2f5c-2c01-465b-a65e-dd3a7830a9ab", "state": "normal", "checkpoint_tso": 449348931263922179, "checkpoint_time": "2024-04-26 19:15:39.404", "error": null } ++ echo '{' '"upstream_id":' 7362132737071396878, '"namespace":' '"default",' '"id":' '"936a2f5c-2c01-465b-a65e-dd3a7830a9ab",' '"state":' '"normal",' '"checkpoint_tso":' 449348931263922179, '"checkpoint_time":' '"2024-04-26' '19:15:39.404",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132737071396878, '"namespace":' '"default",' '"id":' '"936a2f5c-2c01-465b-a65e-dd3a7830a9ab",' '"state":' '"normal",' '"checkpoint_tso":' 449348931263922179, '"checkpoint_time":' '"2024-04-26' '19:15:39.404",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7362132737071396878 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6878ba8c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:9863, start at 2024-04-26 19:15:36.762392191 +0800 CST m=+5.371520187 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:36.771 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:36.739 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:36.739 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6878bc10000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-xz9xx, pid:9937, start at 2024-04-26 19:15:36.852971711 +0800 CST m=+5.378987454 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:36.862 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:36.836 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:36.836 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 5-th time, retry later table processor_delay.t18 exists table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/lossy_ddl Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table processor_resolved_ts_fallback.t2 not exists for 2-th check, retry later table partition_table.finish_mark not exists for 11-th check, retry later table test.finish_mark not exists for 4-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11192.out cli tso query --pd=http://127.0.0.1:2379 table processor_delay.t20 not exists for 2-th check, retry later check diff successfully table partition_table.t not exists for 1-th check, retry later table processor_resolved_ts_fallback.t2 not exists for 3-th check, retry later wait process cdc.test exit for 1-th time... run task successfully Verifying downstream PD is started... Starting Upstream TiKV... Changefeed remove successfully. ID: 650f36b9-3ed3-49a0-8a2e-47842d39de68 CheckpointTs: 449348928970424324 SinkURI: file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-31413?protocol=canal-json&enable-tidb-extension=true check_safepoint_forward http://127.0.0.1:2379 7362132737071396878 449348932115890178 449348928970424324 449348931263922179 TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 2-th time... table partition_table.t not exists for 2-th check, retry later table partition_table.finish_mark not exists for 12-th check, retry later table test.finish_mark exists check table exists success + set +x + tso='449348931954671617 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348931954671617 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:15:43 CST 2024] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1122511227.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully table processor_delay.t20 not exists for 3-th check, retry later wait process cdc.test exit for 3-th time... wait process cdc.test exit for 1-th time... INSERT - Takes(s): 20.0, Count: 4937, OPS: 247.1, Avg(us): 3761, Min(us): 1420, Max(us): 83498, 95th(us): 4000, 99th(us): 67000 Run finished, takes 20.316172574s INSERT - Takes(s): 20.3, Count: 5000, OPS: 246.4, Avg(us): 3774, Min(us): 1420, Max(us): 83498, 95th(us): 4000, 99th(us): 67000 + workdir=/tmp/tidb_cdc_test/big_txn + sink_uri='file:///tmp/tidb_cdc_test/big_txn/storage_test/ticdc-big-txn-test-21441?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/big_txn ++ date + echo '[Fri Apr 26 19:15:44 CST 2024] <<<<<< START storage consumer in big_txn case >>>>>>' [Fri Apr 26 19:15:44 CST 2024] <<<<<< START storage consumer in big_txn case >>>>>> + cd /tmp/tidb_cdc_test/big_txn + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/big_txn + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/big_txn/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/big_txn/storage_test/ticdc-big-txn-test-21441?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table big_txn.usertable not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:15:44 CST 2024] <<<<<< run test case processor_err_chan success! >>>>>> wait process cdc.test exit for 2-th time... run task successfully Changefeed remove successfully. ID: 936a2f5c-2c01-465b-a65e-dd3a7830a9ab CheckpointTs: 449348932378034179 SinkURI: file:///tmp/tidb_cdc_test/gc_safepoint/storage_test/ticdc-gc-safepoint-31413?protocol=canal-json&enable-tidb-extension=true check_safepoint_cleared http://127.0.0.1:2379 7362132737071396878 run task successfully table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' 2 run task successfully killing cdc1: 2570 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 1-th time, retry later wait process cdc.test exit for 3-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... table partition_table.t exists table partition_table.t1 exists table partition_table.finish_mark not exists for 1-th check, retry later table partition_table.finish_mark not exists for 13-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:15:45 CST 2024] <<<<<< run test case charset_gbk success! >>>>>> wait process cdc.test exit for 2-th time... table processor_delay.t20 not exists for 4-th check, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:15:46 CST 2024] <<<<<< run test case gc_safepoint success! >>>>>> table big_txn.usertable not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:46 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/05f96334-3c70-42dd-895f-1da5340e188c {"id":"05f96334-3c70-42dd-895f-1da5340e188c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e01e3f5 05f96334-3c70-42dd-895f-1da5340e188c /tidb/cdc/default/default/upstream/7362132842774682323 {"id":7362132842774682323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/05f96334-3c70-42dd-895f-1da5340e188c {"id":"05f96334-3c70-42dd-895f-1da5340e188c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e01e3f5 05f96334-3c70-42dd-895f-1da5340e188c /tidb/cdc/default/default/upstream/7362132842774682323 {"id":7362132842774682323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/05f96334-3c70-42dd-895f-1da5340e188c {"id":"05f96334-3c70-42dd-895f-1da5340e188c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e01e3f5 05f96334-3c70-42dd-895f-1da5340e188c /tidb/cdc/default/default/upstream/7362132842774682323 {"id":7362132842774682323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11280.out cli changefeed create --start-ts=449348931954671617 '--sink-uri=file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-22764?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 72a48c78-fdef-45fd-8653-f717ae680fc1 Info: {"upstream_id":7362132842774682323,"namespace":"default","id":"72a48c78-fdef-45fd-8653-f717ae680fc1","sink_uri":"file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-22764?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:15:46.939858547+08:00","start_ts":449348931954671617,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348931954671617,"checkpoint_ts":449348931954671617,"checkpoint_time":"2024-04-26 19:15:42.039"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 2-th time, retry later table partition_table.finish_mark not exists for 14-th check, retry later table processor_delay.t20 not exists for 5-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/resourcecontrol + sink_uri='file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-22764?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/resourcecontrol ++ date + echo '[Fri Apr 26 19:15:48 CST 2024] <<<<<< START storage consumer in resourcecontrol case >>>>>>' [Fri Apr 26 19:15:48 CST 2024] <<<<<< START storage consumer in resourcecontrol case >>>>>> + cd /tmp/tidb_cdc_test/resourcecontrol + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/resourcecontrol + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/resourcecontrol/storage_test/ticdc-resourcecontrol-test-22764?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table partition_table.finish_mark not exists for 2-th check, retry later table big_txn.usertable not exists for 3-th check, retry later table resourcecontrol.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... table processor_delay.t20 exists table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... table big_txn.usertable not exists for 4-th check, retry later table resourcecontrol.finish_mark not exists for 2-th check, retry later table partition_table.finish_mark not exists for 3-th check, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:15:51 CST 2024] <<<<<< run test case partition_table success! >>>>>> cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6879975c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13, pid:2219, start at 2024-04-26 19:15:50.919795695 +0800 CST m=+5.794024417 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:50.927 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:50.921 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:50.921 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table processor_delay.t22 not exists for 2-th check, retry later table big_txn.usertable not exists for 5-th check, retry later table resourcecontrol.finish_mark not exists for 3-th check, retry later table partition_table.finish_mark not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6879975c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13, pid:2219, start at 2024-04-26 19:15:50.919795695 +0800 CST m=+5.794024417 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:50.927 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:50.921 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:50.921 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687996900014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13, pid:2290, start at 2024-04-26 19:15:50.867722816 +0800 CST m=+5.679208155 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:50.876 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:50.870 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:50.870 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/lossy_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/lossy_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/lossy_ddl/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t22 not exists for 3-th check, retry later table big_txn.usertable not exists for 6-th check, retry later table partition_table.finish_mark not exists for 5-th check, retry later table resourcecontrol.finish_mark not exists for 4-th check, retry later table processor_delay.t22 not exists for 4-th check, retry later [Fri Apr 26 19:15:55 CST 2024] <<<<<< START cdc server in lossy_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.lossy_ddl.35243526.out server --log-file /tmp/tidb_cdc_test/lossy_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/lossy_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.finish_mark not exists for 6-th check, retry later table big_txn.usertable exists Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table resourcecontrol.finish_mark not exists for 5-th check, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later table processor_delay.t22 not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:58 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/07610e9e-486a-48f5-ad57-b5c7e4e9612e {"id":"07610e9e-486a-48f5-ad57-b5c7e4e9612e","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e3d61f3 07610e9e-486a-48f5-ad57-b5c7e4e9612e /tidb/cdc/default/default/upstream/7362132901513429303 {"id":7362132901513429303,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/07610e9e-486a-48f5-ad57-b5c7e4e9612e {"id":"07610e9e-486a-48f5-ad57-b5c7e4e9612e","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e3d61f3 07610e9e-486a-48f5-ad57-b5c7e4e9612e /tidb/cdc/default/default/upstream/7362132901513429303 {"id":7362132901513429303,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/07610e9e-486a-48f5-ad57-b5c7e4e9612e {"id":"07610e9e-486a-48f5-ad57-b5c7e4e9612e","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e3d61f3 07610e9e-486a-48f5-ad57-b5c7e4e9612e /tidb/cdc/default/default/upstream/7362132901513429303 {"id":7362132901513429303,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table partition_table.finish_mark not exists for 7-th check, retry later Create changefeed successfully! ID: 65c609d1-5537-4489-b6d2-25c09c2bc2cd Info: {"upstream_id":7362132901513429303,"namespace":"default","id":"65c609d1-5537-4489-b6d2-25c09c2bc2cd","sink_uri":"blackhole:","create_time":"2024-04-26T19:15:58.750300213+08:00","start_ts":449348936301805576,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348936301805576,"checkpoint_ts":449348936301805576,"checkpoint_time":"2024-04-26 19:15:58.622"} =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table resourcecontrol.finish_mark exists check diff successfully check diff failed 2-th time, retry later wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... table processor_delay.t22 exists table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:00 CST 2024] <<<<<< run test case resourcecontrol success! >>>>>> table partition_table.finish_mark not exists for 8-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later table processor_delay.t24 not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table partition_table.finish_mark not exists for 9-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) check diff failed 4-th time, retry later [Pipeline] // cache table processor_delay.t24 not exists for 3-th check, retry later [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table partition_table.finish_mark not exists for 10-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... table processor_delay.t24 not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687a6c400004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36, pid:3613, start at 2024-04-26 19:16:04.501951311 +0800 CST m=+5.523868669 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:04.510 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:04.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:04.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687a6c400004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36, pid:3613, start at 2024-04-26 19:16:04.501951311 +0800 CST m=+5.523868669 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:04.510 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:04.496 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:04.496 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687a6cd80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-1sl36, pid:3676, start at 2024-04-26 19:16:04.570704879 +0800 CST m=+5.540585106 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:04.578 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:04.534 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:04.534 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:07 CST 2024] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> table partition_table.finish_mark exists check diff successfully table processor_delay.t24 not exists for 5-th check, retry later wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 1-th time, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 2-th time, retry later [Fri Apr 26 19:16:09 CST 2024] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.49744976.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:16:09 CST 2024] <<<<<< run test case csv_storage_partition_table success! >>>>>> table processor_delay.t24 exists table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ac2f40004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:23265, start at 2024-04-26 19:16:10.049375229 +0800 CST m=+5.453153621 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:10.057 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:10.045 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:10.045 +0800 All versions after safe point can be accessed. (DO NOT EDIT) start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... table processor_delay.t26 not exists for 2-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:12 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0dbbc79b-51d0-478e-add6-824635797113 {"id":"0dbbc79b-51d0-478e-add6-824635797113","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e75544a 0dbbc79b-51d0-478e-add6-824635797113 /tidb/cdc/default/default/upstream/7362132959057243066 {"id":7362132959057243066,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0dbbc79b-51d0-478e-add6-824635797113 {"id":"0dbbc79b-51d0-478e-add6-824635797113","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e75544a 0dbbc79b-51d0-478e-add6-824635797113 /tidb/cdc/default/default/upstream/7362132959057243066 {"id":7362132959057243066,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0dbbc79b-51d0-478e-add6-824635797113 {"id":"0dbbc79b-51d0-478e-add6-824635797113","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e75544a 0dbbc79b-51d0-478e-add6-824635797113 /tidb/cdc/default/default/upstream/7362132959057243066 {"id":7362132959057243066,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.cli.5022.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-26975?protocol=canal-json&enable-tidb-extension=true' [Pipeline] // cache VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ac2f40004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:23265, start at 2024-04-26 19:16:10.049375229 +0800 CST m=+5.453153621 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:10.057 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:10.045 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:10.045 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ac3c40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:23341, start at 2024-04-26 19:16:10.122933466 +0800 CST m=+5.463653946 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:10.131 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:10.097 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:10.097 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [Pipeline] } [Pipeline] // dir Create changefeed successfully! ID: c118baec-f1f0-4a79-82c1-ec5b36dd3480 Info: {"upstream_id":7362132959057243066,"namespace":"default","id":"c118baec-f1f0-4a79-82c1-ec5b36dd3480","sink_uri":"file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-26975?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:16:12.847674518+08:00","start_ts":449348940004327426,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348940004327426,"checkpoint_ts":449348940004327426,"checkpoint_time":"2024-04-26 19:16:12.746"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] } Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ae5100007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:4300, start at 2024-04-26 19:16:12.236796819 +0800 CST m=+5.320184497 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:12.244 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:12.228 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:12.228 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 3-th time, retry later [Pipeline] // stage Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] } table processor_delay.t26 not exists for 3-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... + set +x + workdir=/tmp/tidb_cdc_test/region_merge + sink_uri='file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-26975?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/region_merge ++ date + echo '[Fri Apr 26 19:16:14 CST 2024] <<<<<< START storage consumer in region_merge case >>>>>>' [Fri Apr 26 19:16:14 CST 2024] <<<<<< START storage consumer in region_merge case >>>>>> + cd /tmp/tidb_cdc_test/region_merge + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/region_merge + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/region_merge/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/region_merge/storage_test/ticdc-region-merge-test-26975?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ae5100007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:4300, start at 2024-04-26 19:16:12.236796819 +0800 CST m=+5.320184497 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:12.244 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:12.228 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:12.228 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ae4280013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:4371, start at 2024-04-26 19:16:12.201893628 +0800 CST m=+5.233478331 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:12.209 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:12.170 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:12.170 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.24604.out cli tso query --pd=http://127.0.0.1:2379 split_and_random_merge scale: 20 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t26 not exists for 4-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449348940659425281 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348940659425281 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "threadcount"="10" "requestdistribution"="uniform" "readallfields"="true" "mysql.port"="4000" "mysql.host"="127.0.0.1" "recordcount"="10000" "readproportion"="0" "operationcount"="0" "mysql.db"="move_table" "updateproportion"="0" "mysql.user"="root" "insertproportion"="0" "scanproportion"="0" "workload"="core" "dotransactions"="false" ********************************************** [Fri Apr 26 19:16:17 CST 2024] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.56825684.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t26 exists table processor_delay.t27 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Run finished, takes 2.089340965s INSERT - Takes(s): 2.1, Count: 10000, OPS: 4798.8, Avg(us): 2049, Min(us): 808, Max(us): 44237, 95th(us): 2000, 99th(us): 39000 [Fri Apr 26 19:16:18 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2469224694.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 4-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:20 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcfe0dd2-7919-4ce3-b540-7c7003e4fb54 {"id":"bcfe0dd2-7919-4ce3-b540-7c7003e4fb54","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e8ea9f1 bcfe0dd2-7919-4ce3-b540-7c7003e4fb54 /tidb/cdc/default/default/upstream/7362132991359321793 {"id":7362132991359321793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcfe0dd2-7919-4ce3-b540-7c7003e4fb54 {"id":"bcfe0dd2-7919-4ce3-b540-7c7003e4fb54","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e8ea9f1 bcfe0dd2-7919-4ce3-b540-7c7003e4fb54 /tidb/cdc/default/default/upstream/7362132991359321793 {"id":7362132991359321793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bcfe0dd2-7919-4ce3-b540-7c7003e4fb54 {"id":"bcfe0dd2-7919-4ce3-b540-7c7003e4fb54","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e8ea9f1 bcfe0dd2-7919-4ce3-b540-7c7003e4fb54 /tidb/cdc/default/default/upstream/7362132991359321793 {"id":7362132991359321793,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + workdir=/tmp/tidb_cdc_test/changefeed_pause_resume + sink_uri='file:///tmp/tidb_cdc_test/changefeed_pause_resume/storage_test/ticdc-changefeed-pause-resume-3428?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_pause_resume ++ date + echo '[Fri Apr 26 19:16:20 CST 2024] <<<<<< START storage consumer in changefeed_pause_resume case >>>>>>' [Fri Apr 26 19:16:20 CST 2024] <<<<<< START storage consumer in changefeed_pause_resume case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_pause_resume + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_pause_resume + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_pause_resume/storage_test/ticdc-changefeed-pause-resume-3428?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table changefeed_pause_resume.t1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t28 not exists for 2-th check, retry later *************************** 1. row *************************** count(distinct region_id): 1 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:22 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.24753.out cli changefeed create --start-ts=449348940659425281 '--sink-uri=file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 2d5d679a-4fb6-4478-9d97-790e32516946 Info: {"upstream_id":7362132985381991358,"namespace":"default","id":"2d5d679a-4fb6-4478-9d97-790e32516946","sink_uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:16:22.41236004+08:00","start_ts":449348940659425281,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348940659425281,"checkpoint_ts":449348940659425281,"checkpoint_time":"2024-04-26 19:16:15.245"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node table changefeed_pause_resume.t1 not exists for 2-th check, retry later [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x + workdir=/tmp/tidb_cdc_test/move_table + sink_uri='file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/move_table ++ date + echo '[Fri Apr 26 19:16:23 CST 2024] <<<<<< START storage consumer in move_table case >>>>>>' [Fri Apr 26 19:16:23 CST 2024] <<<<<< START storage consumer in move_table case >>>>>> + cd /tmp/tidb_cdc_test/move_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/move_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/move_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' [Fri Apr 26 19:16:23 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2478224784.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t28 not exists for 3-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b9df00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:3666, start at 2024-04-26 19:16:24.11073426 +0800 CST m=+8.055735920 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:24.133 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:24.110 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:24.110 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b82d4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz, pid:3416, start at 2024-04-26 19:16:22.363938477 +0800 CST m=+8.071544246 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:22.387 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:22.375 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:22.375 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b82d4000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz, pid:3416, start at 2024-04-26 19:16:22.363938477 +0800 CST m=+8.071544246 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:22.387 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:22.375 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:22.375 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b85280002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-4p9zz, pid:3454, start at 2024-04-26 19:16:22.479829195 +0800 CST m=+8.055334959 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:22.501 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:22.474 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:22.474 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table changefeed_pause_resume.t1 not exists for 3-th check, retry later split_and_random_merge scale: 40 table processor_delay.t28 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:27 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/2d5d679a-4fb6-4478-9d97-790e32516946 {UpstreamID:7362132985381991358 Namespace:default ID:2d5d679a-4fb6-4478-9d97-790e32516946 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:22.41236004 +0800 CST StartTs:449348940659425281 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0030cfc20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348942533754883} {CheckpointTs:449348941052641314 MinTableBarrierTs:449348943647866889 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b8b6188d-3568-4c57-9c76-433d1daef538 {"id":"b8b6188d-3568-4c57-9c76-433d1daef538","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e839752 b8b6188d-3568-4c57-9c76-433d1daef538 /tidb/cdc/default/default/changefeed/info/2d5d679a-4fb6-4478-9d97-790e32516946 {"upstream-id":7362132985381991358,"namespace":"default","changefeed-id":"2d5d679a-4fb6-4478-9d97-790e32516946","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:22.41236004+08:00","start-ts":449348940659425281,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348942533754883} /tidb/cdc/default/default/changefeed/status/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":449348941052641314,"min-table-barrier-ts":449348943647866889,"admin-job-type":0} /tidb/cdc/default/default/task/position/159a97c1-e9b3-4c9d-80f0-8d4b988680ba/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b8b6188d-3568-4c57-9c76-433d1daef538/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/2d5d679a-4fb6-4478-9d97-790e32516946 {UpstreamID:7362132985381991358 Namespace:default ID:2d5d679a-4fb6-4478-9d97-790e32516946 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:22.41236004 +0800 CST StartTs:449348940659425281 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0030cfc20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348942533754883} {CheckpointTs:449348941052641314 MinTableBarrierTs:449348943647866889 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b8b6188d-3568-4c57-9c76-433d1daef538 {"id":"b8b6188d-3568-4c57-9c76-433d1daef538","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e839752 b8b6188d-3568-4c57-9c76-433d1daef538 /tidb/cdc/default/default/changefeed/info/2d5d679a-4fb6-4478-9d97-790e32516946 {"upstream-id":7362132985381991358,"namespace":"default","changefeed-id":"2d5d679a-4fb6-4478-9d97-790e32516946","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:22.41236004+08:00","start-ts":449348940659425281,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348942533754883} /tidb/cdc/default/default/changefeed/status/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":449348941052641314,"min-table-barrier-ts":449348943647866889,"admin-job-type":0} /tidb/cdc/default/default/task/position/159a97c1-e9b3-4c9d-80f0-8d4b988680ba/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b8b6188d-3568-4c57-9c76-433d1daef538/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/2d5d679a-4fb6-4478-9d97-790e32516946 {UpstreamID:7362132985381991358 Namespace:default ID:2d5d679a-4fb6-4478-9d97-790e32516946 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:22.41236004 +0800 CST StartTs:449348940659425281 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0030cfc20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348942533754883} {CheckpointTs:449348941052641314 MinTableBarrierTs:449348943647866889 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b8b6188d-3568-4c57-9c76-433d1daef538 {"id":"b8b6188d-3568-4c57-9c76-433d1daef538","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e839752 b8b6188d-3568-4c57-9c76-433d1daef538 /tidb/cdc/default/default/changefeed/info/2d5d679a-4fb6-4478-9d97-790e32516946 {"upstream-id":7362132985381991358,"namespace":"default","changefeed-id":"2d5d679a-4fb6-4478-9d97-790e32516946","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:22.41236004+08:00","start-ts":449348940659425281,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348942533754883} /tidb/cdc/default/default/changefeed/status/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":449348941052641314,"min-table-barrier-ts":449348943647866889,"admin-job-type":0} /tidb/cdc/default/default/task/position/159a97c1-e9b3-4c9d-80f0-8d4b988680ba/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b8b6188d-3568-4c57-9c76-433d1daef538/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key+ grep -q 'etcd info' -path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Fri Apr 26 19:16:27 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.2487924881.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_pause_resume.t1 not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b9df00011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:3666, start at 2024-04-26 19:16:24.11073426 +0800 CST m=+8.055735920 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:24.133 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:24.110 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:24.110 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b9ec80003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:3706, start at 2024-04-26 19:16:24.120980522 +0800 CST m=+7.948883521 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:24.140 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:24.114 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:24.114 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } table processor_delay.t28 exists table processor_delay.t29 not exists for 1-th check, retry later [Pipeline] // withCredentials [Pipeline] } check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 5-th time, retry later [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table changefeed_pause_resume.t1 not exists for 5-th check, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later [Fri Apr 26 19:16:29 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.46614663.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/2d5d679a-4fb6-4478-9d97-790e32516946 {UpstreamID:7362132985381991358 Namespace:default ID:2d5d679a-4fb6-4478-9d97-790e32516946 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:22.41236004 +0800 CST StartTs:449348940659425281 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c78a20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348942533754883} {CheckpointTs:449348941209927884 MinTableBarrierTs:449348944454221833 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/61052886-f6eb-4688-85a7-b30298255c57 {"id":"61052886-f6eb-4688-85a7-b30298255c57","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b8b6188d-3568-4c57-9c76-433d1daef538 {"id":"b8b6188d-3568-4c57-9c76-433d1daef538","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e839752 b8b6188d-3568-4c57-9c76-433d1daef538 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83976b 61052886-f6eb-4688-85a7-b30298255c57 /tidb/cdc/default/default/changefeed/info/2d5d679a-4fb6-4478-9d97-790e32516946 {"upstream-id":7362132985381991358,"namespace":"default","changefeed-id":"2d5d679a-4fb6-4478-9d97-790e32516946","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:22.41236004+08:00","start-ts":449348940659425281,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348942533754883} /tidb/cdc/default/default/changefeed/status/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":449348941209927884,"min-table-barrier-ts":449348944454221833,"admin-job-type":0} /tidb/cdc/default/default/task/position/159a97c1-e9b3-4c9d-80f0-8d4b988680ba/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/61052886-f6eb-4688-85a7-b30298255c57/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b8b6188d-3568-4c57-9c76-433d1daef538/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/2d5d679a-4fb6-4478-9d97-790e32516946 {UpstreamID:7362132985381991358 Namespace:default ID:2d5d679a-4fb6-4478-9d97-790e32516946 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:22.41236004 +0800 CST StartTs:449348940659425281 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c78a20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348942533754883} {CheckpointTs:449348941209927884 MinTableBarrierTs:449348944454221833 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/61052886-f6eb-4688-85a7-b30298255c57 {"id":"61052886-f6eb-4688-85a7-b30298255c57","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b8b6188d-3568-4c57-9c76-433d1daef538 {"id":"b8b6188d-3568-4c57-9c76-433d1daef538","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e839752 b8b6188d-3568-4c57-9c76-433d1daef538 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83976b 61052886-f6eb-4688-85a7-b30298255c57 /tidb/cdc/default/default/changefeed/info/2d5d679a-4fb6-4478-9d97-790e32516946 {"upstream-id":7362132985381991358,"namespace":"default","changefeed-id":"2d5d679a-4fb6-4478-9d97-790e32516946","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:22.41236004+08:00","start-ts":449348940659425281,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348942533754883} /tidb/cdc/default/default/changefeed/status/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":449348941209927884,"min-table-barrier-ts":449348944454221833,"admin-job-type":0} /tidb/cdc/default/default/task/position/159a97c1-e9b3-4c9d-80f0-8d4b988680ba/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/61052886-f6eb-4688-8+ grep -q 'failed to get info:' 5a7-b30298255c57/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b8b6188d-3568-4c57-9c76-433d1daef538/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/2d5d679a-4fb6-4478-9d97-790e32516946 {UpstreamID:7362132985381991358 Namespace:default ID:2d5d679a-4fb6-4478-9d97-790e32516946 SinkURI:file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:22.41236004 +0800 CST StartTs:449348940659425281 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c78a20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348942533754883} {CheckpointTs:449348941209927884 MinTableBarrierTs:449348944454221833 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/159a97c1-e9b3-4c9d-80f0-8d4b988680ba {"id":"159a97c1-e9b3-4c9d-80f0-8d4b988680ba","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/61052886-f6eb-4688-85a7-b30298255c57 {"id":"61052886-f6eb-4688-85a7-b30298255c57","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/b8b6188d-3568-4c57-9c76-433d1daef538 {"id":"b8b6188d-3568-4c57-9c76-433d1daef538","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83970a 159a97c1-e9b3-4c9d-80f0-8d4b988680ba /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e839752 b8b6188d-3568-4c57-9c76-433d1daef538 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e83976b 61052886-f6eb-4688-85a7-b30298255c57 /tidb/cdc/default/default/changefeed/info/2d5d679a-4fb6-4478-9d97-790e32516946 {"upstream-id":7362132985381991358,"namespace":"default","changefeed-id":"2d5d679a-4fb6-4478-9d97-790e32516946","sink-uri":"file:///tmp/tidb_cdc_test/move_table/storage_test/ticdc-move-table-test-19310?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:22.41236004+08:00","start-ts":449348940659425281,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348942533754883} /tidb/cdc/default/default/changefeed/status/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":449348941209927884,"min-table-barrier-ts":449348944454221833,"admin-job-type":0} /tidb/cdc/default/default/task/position/159a97c1-e9b3-4c9d-80f0-8d4b988680ba/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/61052886-f6eb-4688-85a7-b30298255c57/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/b8b6188d-3568-4c57-9c76-433d1daef538/2d5d679a-4fb6-4478-9d97-790e32516946 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132985381991358 {"id":7362132985381991358,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table move_table.usertable not exists for 1-th check, retry later [Fri Apr 26 19:16:30 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.49104912.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 exists table processor_delay.t30 not exists for 2-th check, retry later table move_table.usertable not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:32 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4 {"id":"c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf09f6 c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4 /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4 {"id":"c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf09f6 c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4 /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4 {"id":"c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf09f6 c0be7732-fcd6-4a5b-87d4-2fa8ad778ea4 /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.4703.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true' -c=ddl-manager *************************** 1. row *************************** count(distinct region_id): 1 check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:33 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/014d1369-b280-442d-954d-f9b3b07d2b5f {"id":"014d1369-b280-442d-954d-f9b3b07d2b5f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb1ecf8 014d1369-b280-442d-954d-f9b3b07d2b5f /tidb/cdc/default/default/upstream/7362133022124158550 {"id":7362133022124158550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/014d1369-b280-442d-954d-f9b3b07d2b5f {"id":"014d1369-b280-442d-954d-f9b3b07d2b5f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb1ecf8 014d1369-b280-442d-954d-f9b3b07d2b5f /tidb/cdc/default/default/upstream/7362133022124158550 {"id":7362133022124158550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/014d1369-b280-442d-954d-f9b3b07d2b5f {"id":"014d1369-b280-442d-954d-f9b3b07d2b5f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb1ecf8 014d1369-b280-442d-954d-f9b3b07d2b5f /tidb/cdc/default/default/upstream/7362133022124158550 {"id":7362133022124158550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7362133026512134232,"namespace":"default","id":"ddl-manager","sink_uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:16:33.337742922+08:00","start_ts":449348945359667206,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348945359667206,"checkpoint_ts":449348945359667206,"checkpoint_time":"2024-04-26 19:16:33.175"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + workdir=/tmp/tidb_cdc_test/changefeed_reconstruct + sink_uri='file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-3391?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_reconstruct ++ date + echo '[Fri Apr 26 19:16:33 CST 2024] <<<<<< START storage consumer in changefeed_reconstruct case >>>>>>' [Fri Apr 26 19:16:33 CST 2024] <<<<<< START storage consumer in changefeed_reconstruct case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_reconstruct + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_reconstruct + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-3391?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' ***************** properties ***************** "readallfields"="true" "mysql.user"="root" "workload"="core" "mysql.host"="127.0.0.1" "readproportion"="0" "updateproportion"="0" "requestdistribution"="uniform" "insertproportion"="0" "mysql.db"="changefeed_reconstruct" "mysql.port"="4000" "dotransactions"="false" "recordcount"="50" "operationcount"="0" "scanproportion"="0" "threadcount"="4" ********************************************** Run finished, takes 45.780555ms INSERT - Takes(s): 0.0, Count: 48, OPS: 1435.9, Avg(us): 3655, Min(us): 1839, Max(us): 12684, 95th(us): 13000, 99th(us): 13000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later table move_table.usertable exists go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20230619063055-fd67d94318fd go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/aws/smithy-go v1.13.5 table processor_delay.t30 not exists for 3-th check, retry later go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading go.etcd.io/bbolt v1.3.8 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 + set +x + workdir=/tmp/tidb_cdc_test/ddl_manager + sink_uri='file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_manager ++ date + echo '[Fri Apr 26 19:16:35 CST 2024] <<<<<< START storage consumer in ddl_manager case >>>>>>' [Fri Apr 26 19:16:35 CST 2024] <<<<<< START storage consumer in ddl_manager case >>>>>> + cd /tmp/tidb_cdc_test/ddl_manager + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_manager + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_manager/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 2-th time, retry later table changefeed_reconstruct.usertable not exists for 2-th check, retry later go: downloading github.com/ardielle/ardielle-go v1.5.2 table processor_delay.t30 not exists for 4-th check, retry later split_and_random_merge scale: 80 check diff failed 3-th time, retry later table changefeed_reconstruct.usertable not exists for 3-th check, retry later table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 6-th time, retry later check diff failed 4-th time, retry later table processor_delay.t31 exists table processor_delay.t32 not exists for 1-th check, retry later table changefeed_reconstruct.usertable not exists for 4-th check, retry later check diff failed 5-th time, retry later table processor_delay.t32 not exists for 2-th check, retry later table changefeed_reconstruct.usertable not exists for 5-th check, retry later check diff failed 6-th time, retry later *************************** 1. row *************************** count(distinct region_id): 1 table changefeed_reconstruct.usertable exists check diff successfully table processor_delay.t32 not exists for 3-th check, retry later wait process 4915 exit for 1-th time... wait process 4915 exit for 2-th time... check diff failed 7-th time, retry later wait process 4915 exit for 3-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (4915) - No such process wait process 4915 exit for 4-th time... process 4915 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 26 19:16:46 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.50955097.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t32 not exists for 4-th check, retry later table region_merge.t1 exists check diff failed 8-th time, retry later check diff failed 1-th time, retry later table processor_delay.t32 exists table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:49 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {UpstreamID:7362133022124158550 Namespace:default ID:c0ac7d33-8f70-44ed-bef7-482cb1a902c0 SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-3391?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.457963613 +0800 CST StartTs:449348945395056643 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00015e750 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945421533188} {CheckpointTs:449348948056080387 MinTableBarrierTs:449348949366800388 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348948056080387, checkpointTs: 449348948056080387, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc34f651-8451-47b0-9959-8cfc33c879eb {"id":"bc34f651-8451-47b0-9959-8cfc33c879eb","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb1ed80 bc34f651-8451-47b0-9959-8cfc33c879eb /tidb/cdc/default/default/changefeed/info/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"upstream-id":7362133022124158550,"namespace":"default","changefeed-id":"c0ac7d33-8f70-44ed-bef7-482cb1a902c0","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-3391?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.457963613+08:00","start-ts":449348945395056643,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945421533188} /tidb/cdc/default/default/changefeed/status/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"checkpoint-ts":449348948056080387,"min-table-barrier-ts":449348949366800388,"admin-job-type":0} /tidb/cdc/default/default/task/position/bc34f651-8451-47b0-9959-8cfc33c879eb/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133022124158550 {"id":7362133022124158550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {UpstreamID:7362133022124158550 Namespace:default ID:c0ac7d33-8f70-44ed-bef7-482cb1a902c0 SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-3391?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.457963613 +0800 CST StartTs:449348945395056643 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00015e750 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945421533188} {CheckpointTs:449348948056080387 MinTableBarrierTs:449348949366800388 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348948056080387, checkpointTs: 449348948056080387, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc34f651-8451-47b0-9959-8cfc33c879eb {"id":"bc34f651-8451-47b0-9959-8cfc33c879eb","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb1ed80 bc34f651-8451-47b0-9959-8cfc33c879eb /tidb/cdc/default/default/changefeed/info/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"upstream-id":7362133022124158550,"namespace":"default","changefeed-id":"c0ac7d33-8f70-44ed-bef7-482cb1a902c0","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-3391?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.457963613+08:00","start-ts":449348945395056643,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945421533188} /tidb/cdc/default/default/changefeed/status/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"checkpoint-ts":449348948056080387,"min-table-barrier-ts":449348949366800388,"admin-job-type":0} /tidb/cdc/default/default/task/position/bc34f651-8451-47b0-9959-8cfc33c879eb/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133022124158550 {"id":7362133022124158550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {UpstreamID:7362133022124158550 Namespace:default ID:c0ac7d33-8f70-44ed-bef7-482cb1a902c0 SinkURI:file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-3391?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.457963613 +0800 CST StartTs:449348945395056643 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00015e750 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945421533188} {CheckpointTs:449348948056080387 MinTableBarrierTs:449348949366800388 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348948056080387, checkpointTs: 449348948056080387, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc34f651-8451-47b0-9959-8cfc33c879eb {"id":"bc34f651-8451-47b0-9959-8cfc33c879eb","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb1ed80 bc34f651-8451-47b0-9959-8cfc33c879eb /tidb/cdc/default/default/changefeed/info/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"upstream-id":7362133022124158550,"namespace":"default","changefeed-id":"c0ac7d33-8f70-44ed-bef7-482cb1a902c0","sink-uri":"file:///tmp/tidb_cdc_test/changefeed_reconstruct/storage_test/ticdc-changefeed-reconstruct-3391?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.457963613+08:00","start-ts":449348945395056643,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945421533188} /tidb/cdc/default/default/changefeed/status/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"checkpoint-ts":449348948056080387,"min-table-barrier-ts":449348949366800388,"admin-job-type":0} /tidb/cdc/default/default/task/position/bc34f651-8451-47b0-9959-8cfc33c879eb/c0ac7d33-8f70-44ed-bef7-482cb1a902c0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133022124158550 {"id":7362133022124158550,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id check diff failed 9-th time, retry later wait process 4666 exit for 1-th time... wait process 4666 exit for 2-th time... wait process 4666 exit for 3-th time... wait process 4666 exit for 4-th time... wait process 4666 exit for 5-th time... check diff failed 2-th time, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl can't found finish mark run task failed 7-th time, retry later wait process 4666 exit for 6-th time... table processor_delay.t34 not exists for 2-th check, retry later "id": "bc34f651-8451-47b0-9959-8cfc33c879eb", "cluster-id": "default" run task successfully /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (4666) - No such process wait process 4666 exit for 7-th time... process 4666 already exit [Fri Apr 26 19:16:51 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.47754777.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully check diff failed 3-th time, retry later table processor_delay.t34 not exists for 3-th check, retry later capture_id: bc34f651-8451-47b0-9959-8cfc33c879eb check_processor_table_count http://127.0.0.1:2379 c0ac7d33-8f70-44ed-bef7-482cb1a902c0 bc34f651-8451-47b0-9959-8cfc33c879eb 1 run task successfully check_processor_table_count http://127.0.0.1:2379 c0ac7d33-8f70-44ed-bef7-482cb1a902c0 bc34f651-8451-47b0-9959-8cfc33c879eb 0 check diff failed 1-th time, retry later table count 1 does equal to expected count 0 run task failed 1-th time, retry later check diff failed 4-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > table processor_delay.t34 not exists for 4-th check, retry later < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:55 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362133026512134232 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.337742922 +0800 CST StartTs:449348945359667206 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003658510 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945398988803} {CheckpointTs:449348946146099218 MinTableBarrierTs:449348946146099218 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76335d91-1d25-4e99-a24c-f1631af10420 {"id":"76335d91-1d25-4e99-a24c-f1631af10420","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf0bc7 76335d91-1d25-4e99-a24c-f1631af10420 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362133026512134232,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.337742922+08:00","start-ts":449348945359667206,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945398988803} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348946146099218,"min-table-barrier-ts":449348946146099218,"admin-job-type":0} /tidb/cdc/default/default/task/position/76335d91-1d25-4e99-a24c-f1631af10420/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362133026512134232 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.337742922 +0800 CST StartTs:449348945359667206 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003658510 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945398988803} {CheckpointTs:449348946146099218 MinTableBarrierTs:449348946146099218 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76335d91-1d25-4e99-a24c-f1631af10420 {"id":"76335d91-1d25-4e99-a24c-f1631af10420","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf0bc7 76335d91-1d25-4e99-a24c-f1631af10420 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362133026512134232,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.337742922+08:00","start-ts":449348945359667206,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945398988803} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348946146099218,"min-table-barrier-ts":449348946146099218,"admin-job-type":0} /tidb/cdc/default/default/task/position/76335d91-1d25-4e99-a24c-f1631af10420/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362133026512134232 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.337742922 +0800 CST StartTs:449348945359667206 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003658510 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945398988803} {CheckpointTs:449348946146099218 MinTableBarrierTs:449348946146099218 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76335d91-1d25-4e99-a24c-f1631af10420 {"id":"76335d91-1d25-4e99-a24c-f1631af10420","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf0bc7 76335d91-1d25-4e99-a24c-f1631af10420 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362133026512134232,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.337742922+08:00","start-ts":449348945359667206,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945398988803} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348946146099218,"min-table-barrier-ts":449348946146099218,"admin-job-type":0} /tidb/cdc/default/default/task/position/76335d91-1d25-4e99-a24c-f1631af10420/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:16:55 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.48314833.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > check diff failed 2-th time, retry later check_processor_table_count http://127.0.0.1:2379 c0ac7d33-8f70-44ed-bef7-482cb1a902c0 bc34f651-8451-47b0-9959-8cfc33c879eb 0 run task successfully table processor_delay.t34 not exists for 5-th check, retry later ***************** properties ***************** "threadcount"="4" "mysql.host"="127.0.0.1" "mysql.db"="changefeed_reconstruct" "readallfields"="true" "mysql.user"="root" "workload"="core" "operationcount"="0" "readproportion"="0" "dotransactions"="false" "recordcount"="50" "updateproportion"="0" "scanproportion"="0" "mysql.port"="4000" "insertproportion"="0" "requestdistribution"="uniform" ********************************************** Run finished, takes 61.960006ms INSERT - Takes(s): 0.0, Count: 48, OPS: 1011.2, Avg(us): 4664, Min(us): 2045, Max(us): 13574, 95th(us): 14000, 99th(us): 14000 table changefeed_reconstruct.usertable exists check diff successfully wait process cdc.test exit for 1-th time... check diff failed 1-th time, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:57 CST 2024] <<<<<< run test case region_merge success! >>>>>> check diff failed 3-th time, retry later [2024/04/26 19:16:55.830 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/04/26 19:16:55.834 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/04/26 19:16:55.835 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=159a97c1-e9b3-4c9d-80f0-8d4b988680ba] [2024/04/26 19:16:55.835 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/04/26 19:16:55.835 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/04/26 19:16:56.033 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=2d5d679a-4fb6-4478-9d97-790e32516946] [captureID=159a97c1-e9b3-4c9d-80f0-8d4b988680ba] [processorDetail="{\"table_ids\":[104,106]}"] [2024/04/26 19:16:56.233 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=2d5d679a-4fb6-4478-9d97-790e32516946] [captureID=61052886-f6eb-4688-85a7-b30298255c57] [processorDetail="{\"table_ids\":[]}"] [2024/04/26 19:16:56.433 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=2d5d679a-4fb6-4478-9d97-790e32516946] [captureID=b8b6188d-3568-4c57-9c76-433d1daef538] [processorDetail="{\"table_ids\":[]}"] [2024/04/26 19:16:56.433 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"159a97c1-e9b3-4c9d-80f0-8d4b988680ba\":[{\"ID\":104,\"Changefeed\":\"2d5d679a-4fb6-4478-9d97-790e32516946\"},{\"ID\":106,\"Changefeed\":\"2d5d679a-4fb6-4478-9d97-790e32516946\"}],\"61052886-f6eb-4688-85a7-b30298255c57\":[],\"b8b6188d-3568-4c57-9c76-433d1daef538\":[]}"] [2024/04/26 19:16:56.433 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=2d5d679a-4fb6-4478-9d97-790e32516946&target-cp-id=b8b6188d-3568-4c57-9c76-433d1daef538&table-id=104"] [2024/04/26 19:16:56.483 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=104] [2024/04/26 19:16:56.483 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=2d5d679a-4fb6-4478-9d97-790e32516946&target-cp-id=b8b6188d-3568-4c57-9c76-433d1daef538&table-id=106"] [2024/04/26 19:16:56.533 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=106] [2024/04/26 19:16:56.533 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=159a97c1-e9b3-4c9d-80f0-8d4b988680ba] [targetCapture=b8b6188d-3568-4c57-9c76-433d1daef538] table move_table.check1 exists check diff successfully table processor_delay.t34 exists table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:58 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362133026512134232 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.337742922 +0800 CST StartTs:449348945359667206 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003658510 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945398988803} {CheckpointTs:449348946146099218 MinTableBarrierTs:449348946146099218 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76335d91-1d25-4e99-a24c-f1631af10420 {"id":"76335d91-1d25-4e99-a24c-f1631af10420","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf0bc7 76335d91-1d25-4e99-a24c-f1631af10420 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362133026512134232,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.337742922+08:00","start-ts":449348945359667206,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945398988803} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348946290278409,"min-table-barrier-ts":449348946290278409,"admin-job-type":0} /tidb/cdc/default/default/task/position/76335d91-1d25-4e99-a24c-f1631af10420/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362133026512134232 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.337742922 +0800 CST StartTs:449348945359667206 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003658510 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945398988803} {CheckpointTs:449348946146099218 MinTableBarrierTs:449348946146099218 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76335d91-1d25-4e99-a24c-f1631af10420 {"id":"76335d91-1d25-4e99-a24c-f1631af10420","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf0bc7 76335d91-1d25-4e99-a24c-f1631af10420 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362133026512134232,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.337742922+08:00","start-ts":449348945359667206,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945398988803} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348946290278409,"min-table-barrier-ts":449348946290278409,"admin-job-type":0} /tidb/cdc/default/default/task/position/76335d91-1d25-4e99-a24c-f1631af10420/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362133026512134232 Namespace:default ID:ddl-manager SinkURI:file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:16:33.337742922 +0800 CST StartTs:449348945359667206 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003658510 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348945398988803} {CheckpointTs:449348946146099218 MinTableBarrierTs:449348946146099218 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/76335d91-1d25-4e99-a24c-f1631af10420 {"id":"76335d91-1d25-4e99-a24c-f1631af10420","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eaf0bc7 76335d91-1d25-4e99-a24c-f1631af10420 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362133026512134232,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"file:///tmp/tidb_cdc_test/ddl_manager/storage_test/ticdc-ddl-mamager-test-26833?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:16:33.337742922+08:00","start-ts":449348945359667206,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348945398988803} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348946290278409,"min-table-barrier-ts":449348946290278409,"admin-job-type":0} /tidb/cdc/default/default/task/position/76335d91-1d25-4e99-a24c-f1631af10420/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133026512134232 {"id":7362133026512134232,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later check diff failed 2-th time, retry later check diff failed 4-th time, retry later [2024/04/26 19:17:00.712 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/04/26 19:17:00.714 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/04/26 19:17:00.714 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=159a97c1-e9b3-4c9d-80f0-8d4b988680ba] [2024/04/26 19:17:00.714 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/04/26 19:17:00.715 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] table ddl_manager.finish_mark not exists for 2-th check, retry later [2024/04/26 19:17:00.883 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=2d5d679a-4fb6-4478-9d97-790e32516946] [captureID=159a97c1-e9b3-4c9d-80f0-8d4b988680ba] [processorDetail="{\"table_ids\":[]}"] table processor_delay.t36 not exists for 2-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 3-th time, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 4-th time, retry later [2024/04/26 19:17:01.083 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=2d5d679a-4fb6-4478-9d97-790e32516946] [captureID=61052886-f6eb-4688-85a7-b30298255c57] [processorDetail="{\"table_ids\":[]}"] [2024/04/26 19:17:01.282 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=2d5d679a-4fb6-4478-9d97-790e32516946] [captureID=b8b6188d-3568-4c57-9c76-433d1daef538] [processorDetail="{\"table_ids\":[106,108]}"] [2024/04/26 19:17:01.283 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"159a97c1-e9b3-4c9d-80f0-8d4b988680ba\":[],\"61052886-f6eb-4688-85a7-b30298255c57\":[],\"b8b6188d-3568-4c57-9c76-433d1daef538\":[{\"ID\":106,\"Changefeed\":\"2d5d679a-4fb6-4478-9d97-790e32516946\"},{\"ID\":108,\"Changefeed\":\"2d5d679a-4fb6-4478-9d97-790e32516946\"}]}"] [2024/04/26 19:17:01.283 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=2d5d679a-4fb6-4478-9d97-790e32516946&target-cp-id=61052886-f6eb-4688-85a7-b30298255c57&table-id=106"] [2024/04/26 19:17:01.333 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=106] [2024/04/26 19:17:01.333 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=2d5d679a-4fb6-4478-9d97-790e32516946&target-cp-id=61052886-f6eb-4688-85a7-b30298255c57&table-id=108"] [2024/04/26 19:17:01.383 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=108] [2024/04/26 19:17:01.383 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=b8b6188d-3568-4c57-9c76-433d1daef538] [targetCapture=61052886-f6eb-4688-85a7-b30298255c57] check diff failed 1-th time, retry later check diff failed 3-th time, retry later check diff failed 5-th time, retry later table ddl_manager.finish_mark not exists for 3-th check, retry later table processor_delay.t36 not exists for 3-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 5-th time, retry later check diff successfully table move_table.check2 not exists for 1-th check, retry later check diff failed 6-th time, retry later table big_txn.usertable1 exists table processor_delay.t36 not exists for 4-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 4-th check, retry later check_lossy_ddl /tmp/tidb_cdc_test/lossy_ddl [2024/04/26 19:16:02.369 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348936393555975,\"commit-ts\":449348936393555978,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example1\",\"tbl-id\":104,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:04.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348936498151430,\"commit-ts\":449348936498151433,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example2\",\"tbl-id\":107,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:06.369 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348936590163985,\"commit-ts\":449348936590163988,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example3\",\"tbl-id\":110,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":7,\"Flag\":128,\"Flen\":19,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":7,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":\"2023-04-19 11:48:00\",\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:08.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348936694759443,\"commit-ts\":449348936694759446,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example4\",\"tbl-id\":113,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":15,\"charset\":\"utf8mb4\",\"collation\":\"utf8mb4_bin\",\"flag\":64,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\",\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:10.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348936799617042,\"commit-ts\":449348936799617046,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example5\",\"tbl-id\":116,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":-1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:12.369 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348936891367442,\"commit-ts\":449348936891367445,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example6\",\"tbl-id\":119,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":-1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:14.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348936983117839,\"commit-ts\":449348936983117842,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example7\",\"tbl-id\":122,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8\",\"Collate\":\"utf8_general_ci\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":15,\"charset\":\"utf8\",\"collation\":\"utf8_general_ci\",\"flag\":64,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\",\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:16.369 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937022439444,\"commit-ts\":449348937022439447,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example8\",\"tbl-id\":126,\"is-partition\":true},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":-1,\"default\":null},{\"name\":\"b\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":-1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:16.369 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937035546626,\"commit-ts\":449348937035546628,\"row-id\":2,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example8\",\"tbl-id\":127,\"is-partition\":true},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:18.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937101082640,\"commit-ts\":449348937101082643,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example9\",\"tbl-id\":130,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":15,\"charset\":\"utf8mb4\",\"collation\":\"utf8mb4_bin\",\"flag\":64,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\",\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:20.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937153511435,\"commit-ts\":449348937153511438,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example10\",\"tbl-id\":133,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":5,\"Flag\":0,\"Flen\":22,\"Decimal\":-1,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":5,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:22.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937258369027,\"commit-ts\":449348937258369030,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example11\",\"tbl-id\":136,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":8,\"Flag\":4099,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":8,\"Flag\":0,\"Flen\":20,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":8,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":8,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:24.369 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937350119443,\"commit-ts\":449348937350119446,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example12\",\"tbl-id\":139,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":251,\"Flag\":0,\"Flen\":4294967295,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":251,\"charset\":\"utf8mb4\",\"collation\":\"utf8mb4_bin\",\"flag\":64,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\",\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:26.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937454977034,\"commit-ts\":449348937454977038,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example13\",\"tbl-id\":142,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":247,\"Flag\":0,\"Flen\":1,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":[\"a\",\"b\",\"c\"],\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":247,\"charset\":\"utf8mb4\",\"collation\":\"utf8mb4_bin\",\"flag\":64,\"value\":1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:28.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937546727443,\"commit-ts\":449348937546727450,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example14\",\"tbl-id\":145,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":15,\"Flag\":0,\"Flen\":256,\"Decimal\":0,\"Charset\":\"utf8mb4\",\"Collate\":\"utf8mb4_bin\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":15,\"charset\":\"utf8mb4\",\"collation\":\"utf8mb4_bin\",\"flag\":64,\"value\":\"MjAyMy0wNC0xOSAxMTo0ODowMA==\",\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:30.368 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937651585038,\"commit-ts\":449348937651585041,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example15\",\"tbl-id\":148,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}},{\"ID\":2,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":0,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null},{\"name\":\"b\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":65,\"value\":-1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:32.469 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937756442631,\"commit-ts\":449348937756442634,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example16\",\"tbl-id\":152,\"is-partition\":true},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"a\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:34.469 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937769549840,\"commit-ts\":449348937769549845,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example17\",\"tbl-id\":156,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"a\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":18,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:36.468 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937835085840,\"commit-ts\":449348937835085843,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example18\",\"tbl-id\":160,\"is-partition\":true},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:37.469 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937835085840,\"commit-ts\":449348937835085843,\"row-id\":2,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example18\",\"tbl-id\":163,\"is-partition\":true},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1977,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:37.470 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348937835085840,\"commit-ts\":449348937835085843,\"row-id\":3,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example18\",\"tbl-id\":165,\"is-partition\":true},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1999,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:43.469 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348938044801040,\"commit-ts\":449348938044801041,\"row-id\":4,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example18\",\"tbl-id\":172,\"is-partition\":true},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"id\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":2003,\"default\":null}],\"pre-columns\":null}"] [2024/04/26 19:16:47.469 +08:00] [DEBUG] [black_hole_dml_sink.go:45] ["BlackHoleSink: WriteEvents"] [row="{\"start-ts\":449348938542874639,\"commit-ts\":449348938542874642,\"row-id\":1,\"table\":{\"db-name\":\"lossy_ddl\",\"tbl-name\":\"example19\",\"tbl-id\":180,\"is-partition\":false},\"column-infos\":[{\"ID\":1,\"IsPKHandle\":false,\"VirtualGenCol\":false,\"Ft\":{\"Tp\":3,\"Flag\":4099,\"Flen\":11,\"Decimal\":0,\"Charset\":\"binary\",\"Collate\":\"binary\",\"Elems\":null,\"ElemsIsBinaryLit\":null,\"Array\":false}}],\"columns\":[{\"name\":\"a\",\"type\":3,\"charset\":\"binary\",\"collation\":\"binary\",\"flag\":27,\"value\":1,\"default\":null}],\"pre-columns\":null}"] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:17:04 CST 2024] <<<<<< run test case lossy_ddl success! >>>>>> wait process cdc.test exit for 2-th time... check data successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 3-th time... table move_table.check2 not exists for 2-th check, retry later check diff failed 7-th time, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:17:06 CST 2024] <<<<<< run test case changefeed_reconstruct success! >>>>>> table processor_delay.t36 not exists for 5-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:17:06 CST 2024] <<<<<< run test case ddl_puller_lag success! >>>>>> table ddl_manager.finish_mark not exists for 5-th check, retry later table processor_delay.t36 exists table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later table move_table.check2 not exists for 3-th check, retry later check diff failed 8-th time, retry later table ddl_manager.finish_mark not exists for 6-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:17:09 CST 2024] <<<<<< run test case common_1 success! >>>>>> table move_table.check2 not exists for 4-th check, retry later check diff failed 9-th time, retry later table ddl_manager.finish_mark not exists for 7-th check, retry later table processor_delay.t38 not exists for 2-th check, retry later table move_table.check2 not exists for 5-th check, retry later check diff successfully \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } table ddl_manager.finish_mark not exists for 8-th check, retry later [Pipeline] // timeout [Pipeline] } table processor_delay.t38 not exists for 3-th check, retry later [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table move_table.check2 exists check diff successfully check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 9-th check, retry later wait process cdc.test exit for 1-th time... table processor_delay.t38 not exists for 4-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... check diff failed 2-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:17:16 CST 2024] <<<<<< run test case move_table success! >>>>>> table processor_delay.t38 not exists for 5-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/storage_csv_update Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 10-th check, retry later table processor_delay.t38 exists table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later check diff failed 3-th time, retry later Verifying downstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 11-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 12-th check, retry later table processor_delay.t40 not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later table processor_delay.t40 not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 13-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t40 not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... table ddl_manager.finish_mark not exists for 14-th check, retry later check diff failed 6-th time, retry later Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... table processor_delay.t40 not exists for 5-th check, retry later check diff failed 7-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 15-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687f6d740009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13, pid:4628, start at 2024-04-26 19:17:26.504610367 +0800 CST m=+5.270042578 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:26.513 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:26.493 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:26.493 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table processor_delay.t40 exists table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later check diff failed 8-th time, retry later table ddl_manager.finish_mark not exists for 16-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687f6d740009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13, pid:4628, start at 2024-04-26 19:17:26.504610367 +0800 CST m=+5.270042578 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:26.513 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:26.493 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:26.493 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687f6f2c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-z9x13, pid:4705, start at 2024-04-26 19:17:26.647202106 +0800 CST m=+5.318354109 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:26.654 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:26.653 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:26.653 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/storage_csv_update/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/storage_csv_update/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/storage_csv_update/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table processor_delay.t42 not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 17-th check, retry later check diff failed 9-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:17:31 CST 2024] <<<<<< START cdc server in storage_csv_update case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_csv_update.59845986.out server --log-file /tmp/tidb_cdc_test/storage_csv_update/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/storage_csv_update/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t42 not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 18-th check, retry later check diff successfully start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > table processor_delay.t42 not exists for 4-th check, retry later < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:34 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88f92d95-d366-4277-8d59-0d276c779bc2 {"id":"88f92d95-d366-4277-8d59-0d276c779bc2","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fb6cff2 88f92d95-d366-4277-8d59-0d276c779bc2 /tidb/cdc/default/default/upstream/7362133316135074373 {"id":7362133316135074373,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88f92d95-d366-4277-8d59-0d276c779bc2 {"id":"88f92d95-d366-4277-8d59-0d276c779bc2","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fb6cff2 88f92d95-d366-4277-8d59-0d276c779bc2 /tidb/cdc/default/default/upstream/7362133316135074373 {"id":7362133316135074373,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/88f92d95-d366-4277-8d59-0d276c779bc2 {"id":"88f92d95-d366-4277-8d59-0d276c779bc2","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fb6cff2 88f92d95-d366-4277-8d59-0d276c779bc2 /tidb/cdc/default/default/upstream/7362133316135074373 {"id":7362133316135074373,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.storage_csv_update.cli.6033.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml table ddl_manager.finish_mark not exists for 19-th check, retry later check diff failed 1-th time, retry later Create changefeed successfully! ID: 12c52b40-3b2e-46ee-9039-9f04f2f3a160 Info: {"upstream_id":7362133316135074373,"namespace":"default","id":"12c52b40-3b2e-46ee-9039-9f04f2f3a160","sink_uri":"file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:17:35.182254302+08:00","start_ts":449348961577730054,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"csv","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348961577730054,"checkpoint_ts":449348961577730054,"checkpoint_time":"2024-04-26 19:17:35.042"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t42 not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 20-th check, retry later + set +x check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688019d00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:6209, start at 2024-04-26 19:17:37.533469534 +0800 CST m=+7.802482285 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:37.568 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:37.575 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:37.575 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t42 exists table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 21-th check, retry later check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688019d00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:6209, start at 2024-04-26 19:17:37.533469534 +0800 CST m=+7.802482285 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:37.568 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:37.575 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:37.575 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68801c1c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-3vl80, pid:6253, start at 2024-04-26 19:17:37.717404609 +0800 CST m=+7.853035656 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:37.734 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:37.721 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:37.721 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688014d40016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:6595, start at 2024-04-26 19:17:37.233179418 +0800 CST m=+5.272574661 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:37.243 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:37.205 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:37.205 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688014d40016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:6595, start at 2024-04-26 19:17:37.233179418 +0800 CST m=+5.272574661 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:37.243 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:37.205 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:37.205 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688014c00014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:6676, start at 2024-04-26 19:17:37.241247643 +0800 CST m=+5.207809456 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:37.248 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:37.249 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:37.249 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t44 not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 22-th check, retry later check diff failed 4-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.7502.out cli tso query --pd=http://127.0.0.1:2379 [Fri Apr 26 19:17:42 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.79387940.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t44 not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 23-th check, retry later check diff failed 5-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688068140012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:29470, start at 2024-04-26 19:17:42.556829168 +0800 CST m=+5.124965636 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:42.563 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:42.533 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:42.533 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688068140012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:29470, start at 2024-04-26 19:17:42.556829168 +0800 CST m=+5.124965636 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:42.563 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:42.533 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:42.533 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688069900001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-9mq33, pid:29562, start at 2024-04-26 19:17:42.628819692 +0800 CST m=+5.103512896 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:42.636 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:42.628 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:42.628 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_delay.t44 not exists for 4-th check, retry later check diff failed 6-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:45 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/723932e5-6eae-4596-989f-6fbca90a0e6c {"id":"723932e5-6eae-4596-989f-6fbca90a0e6c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdaccf3 723932e5-6eae-4596-989f-6fbca90a0e6c /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/723932e5-6eae-4596-989f-6fbca90a0e6c {"id":"723932e5-6eae-4596-989f-6fbca90a0e6c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdaccf3 723932e5-6eae-4596-989f-6fbca90a0e6c /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/723932e5-6eae-4596-989f-6fbca90a0e6c {"id":"723932e5-6eae-4596-989f-6fbca90a0e6c","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdaccf3 723932e5-6eae-4596-989f-6fbca90a0e6c /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.7995.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true' -c=ddl-only-block-related-table Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7362133359178103920,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:17:45.83662699+08:00","start_ts":449348964372971529,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348964372971529,"checkpoint_ts":449348964372971529,"checkpoint_time":"2024-04-26 19:17:45.705"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Fri Apr 26 19:17:45 CST 2024] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.3080230804.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t44 not exists for 5-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/ddl_only_block_related_table + sink_uri='file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/ddl_only_block_related_table ++ date + echo '[Fri Apr 26 19:17:47 CST 2024] <<<<<< START storage consumer in ddl_only_block_related_table case >>>>>>' [Fri Apr 26 19:17:47 CST 2024] <<<<<< START storage consumer in ddl_only_block_related_table case >>>>>> + cd /tmp/tidb_cdc_test/ddl_only_block_related_table + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/ddl_only_block_related_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' + set +x + tso='449348963657318402 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348963657318402 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x check diff failed 7-th time, retry later ***************** properties ***************** "operationcount"="0" "updateproportion"="0" "mysql.host"="127.0.0.1" "mysql.user"="root" "insertproportion"="0" "dotransactions"="false" "threadcount"="2" "requestdistribution"="uniform" "readproportion"="0" "recordcount"="10" "scanproportion"="0" "mysql.db"="multi_capture_1" "workload"="core" "mysql.port"="4000" "readallfields"="true" ********************************************** Run finished, takes 23.362942ms INSERT - Takes(s): 0.0, Count: 10, OPS: 803.2, Avg(us): 4521, Min(us): 2340, Max(us): 11028, 95th(us): 12000, 99th(us): 12000 table ddl_manager.finish_mark not exists for 24-th check, retry later ***************** properties ***************** "mysql.port"="4000" "scanproportion"="0" "requestdistribution"="uniform" "mysql.db"="multi_capture_2" "dotransactions"="false" "insertproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "readallfields"="true" "operationcount"="0" "recordcount"="10" "threadcount"="2" "readproportion"="0" "updateproportion"="0" "mysql.user"="root" ********************************************** Run finished, takes 23.760205ms INSERT - Takes(s): 0.0, Count: 10, OPS: 757.2, Avg(us): 4420, Min(us): 2500, Max(us): 10387, 95th(us): 11000, 99th(us): 11000 ***************** properties ***************** "scanproportion"="0" "recordcount"="10" "readproportion"="0" "updateproportion"="0" "mysql.db"="multi_capture_3" "workload"="core" "mysql.user"="root" "readallfields"="true" "mysql.host"="127.0.0.1" "dotransactions"="false" "mysql.port"="4000" "operationcount"="0" "threadcount"="2" "insertproportion"="0" "requestdistribution"="uniform" ********************************************** Run finished, takes 25.845979ms INSERT - Takes(s): 0.0, Count: 10, OPS: 703.4, Avg(us): 4973, Min(us): 2292, Max(us): 11512, 95th(us): 12000, 99th(us): 12000 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:48 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/14fd3cc5-b695-4c16-8b52-29bd234fdaeb {"id":"14fd3cc5-b695-4c16-8b52-29bd234fdaeb","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ff60ff1 14fd3cc5-b695-4c16-8b52-29bd234fdaeb /tidb/cdc/default/default/upstream/7362133384313381040 {"id":7362133384313381040,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/14fd3cc5-b695-4c16-8b52-29bd234fdaeb {"id":"14fd3cc5-b695-4c16-8b52-29bd234fdaeb","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ff60ff1 14fd3cc5-b695-4c16-8b52-29bd234fdaeb /tidb/cdc/default/default/upstream/7362133384313381040 {"id":7362133384313381040,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/14fd3cc5-b695-4c16-8b52-29bd234fdaeb {"id":"14fd3cc5-b695-4c16-8b52-29bd234fdaeb","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ff60ff1 14fd3cc5-b695-4c16-8b52-29bd234fdaeb /tidb/cdc/default/default/upstream/7362133384313381040 {"id":7362133384313381040,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: ebdbd783-79b8-4637-a5d0-24fa15d12eb6 Info: {"upstream_id":7362133384313381040,"namespace":"default","id":"ebdbd783-79b8-4637-a5d0-24fa15d12eb6","sink_uri":"file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-11235?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:17:49.084219153+08:00","start_ts":449348965232279553,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348965232279553,"checkpoint_ts":449348965232279553,"checkpoint_time":"2024-04-26 19:17:48.983"} + workdir=/tmp/tidb_cdc_test/autorandom + sink_uri='file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-11235?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/autorandom ++ date + echo '[Fri Apr 26 19:17:49 CST 2024] <<<<<< START storage consumer in autorandom case >>>>>>' [Fri Apr 26 19:17:49 CST 2024] <<<<<< START storage consumer in autorandom case >>>>>> + cd /tmp/tidb_cdc_test/autorandom + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/autorandom + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/autorandom/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/autorandom/storage_test/ticdc-autorandom-test-11235?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table processor_delay.t44 exists table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later table autorandom_test.table_a not exists for 1-th check, retry later ***************** properties ***************** "mysql.port"="4000" "updateproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "mysql.db"="multi_capture_4" "readproportion"="0" "readallfields"="true" "insertproportion"="0" "recordcount"="10" "workload"="core" "mysql.host"="127.0.0.1" "scanproportion"="0" "operationcount"="0" "threadcount"="2" "dotransactions"="false" ********************************************** Run finished, takes 32.66215ms INSERT - Takes(s): 0.0, Count: 10, OPS: 529.8, Avg(us): 6283, Min(us): 3374, Max(us): 13674, 95th(us): 14000, 99th(us): 14000 [Fri Apr 26 19:17:49 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.76077609.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 25-th check, retry later check diff failed 8-th time, retry later table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later table processor_delay.t46 not exists for 2-th check, retry later table autorandom_test.table_a not exists for 2-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later check diff successfully table ddl_manager.finish_mark not exists for 26-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:52 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:17:52 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.76447646.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t46 not exists for 3-th check, retry later table autorandom_test.table_a not exists for 3-th check, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 27-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later table processor_delay.t46 not exists for 4-th check, retry later table autorandom_test.table_a not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:55 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/e3f6e174-a4ce-427c-a351-407cd7d55752 {"id":"e3f6e174-a4ce-427c-a351-407cd7d55752","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd5657a e3f6e174-a4ce-427c-a351-407cd7d55752 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/e3f6e174-a4ce-427c-a351-407cd7d55752 {"id":"e3f6e174-a4ce-427c-a351-407cd7d55752","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd5657a e3f6e174-a4ce-427c-a351-407cd7d55752 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/e3f6e174-a4ce-427c-a351-407cd7d55752 {"id":"e3f6e174-a4ce-427c-a351-407cd7d55752","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd5657a e3f6e174-a4ce-427c-a351-407cd7d55752 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:17:55 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.76897691.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8303; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 28-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 4-th check, retry later check diff failed 2-th time, retry later table processor_delay.t46 not exists for 5-th check, retry later table autorandom_test.table_a not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 29-th check, retry later table ddl_only_block_related_table.finish_mark not exists for 5-th check, retry later check diff failed 3-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8303 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:58 GMT < Content-Length: 1144 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7 {"id":"0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7","address":"127.0.0.1:8303","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/e3f6e174-a4ce-427c-a351-407cd7d55752 {"id":"e3f6e174-a4ce-427c-a351-407cd7d55752","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd5657a e3f6e174-a4ce-427c-a351-407cd7d55752 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56589 0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7 {"id":"0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7","address":"127.0.0.1:8303","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/e3f6e174-a4ce-427c-a351-407cd7d55752 {"id":"e3f6e174-a4ce-427c-a351-407cd7d55752","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd5657a e3f6e174-a4ce-427c-a351-407cd7d55752 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56589 0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7 {"id":"0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7","address":"127.0.0.1:8303","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 {"id":"6cdfd36f-d9a2-4ef7-91d1-8069e6205c48","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/e3f6e174-a4ce-427c-a351-407cd7d55752 {"id":"e3f6e174-a4ce-427c-a351-407cd7d55752","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56551 6cdfd36f-d9a2-4ef7-91d1-8069e6205c48 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd5657a e3f6e174-a4ce-427c-a351-407cd7d55752 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fd56589 0c98b7bb-b8fe-40cc-bff5-c9a16d7c82f7 /tidb/cdc/default/default/upstream/7362133346669113224 {"id":7362133346669113224,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.7727.out cli changefeed create --start-ts=449348963657318402 '--sink-uri=file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-21141?protocol=canal-json&enable-tidb-extension=true' --server=127.0.0.1:8301 Create changefeed successfully! ID: 97ce5736-4032-4672-a25f-11ab163c3f2e Info: {"upstream_id":7362133346669113224,"namespace":"default","id":"97ce5736-4032-4672-a25f-11ab163c3f2e","sink_uri":"file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-21141?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:17:59.11963312+08:00","start_ts":449348963657318402,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449348963657318402,"checkpoint_ts":449348963657318402,"checkpoint_time":"2024-04-26 19:17:42.975"} PASS table processor_delay.t46 exists table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later coverage: 2.5% of statements in github.com/pingcap/tiflow/... table autorandom_test.table_a exists check diff successfully table ddl_manager.finish_mark not exists for 30-th check, retry later table ddl_only_block_related_table.finish_mark exists wait process cdc.test exit for 1-th time... check diff failed 4-th time, retry later wait process 7943 exit for 1-th time... wait process cdc.test exit for 2-th time... wait process 7943 exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:18:00 CST 2024] <<<<<< run test case autorandom success! >>>>>> table processor_delay.t48 not exists for 2-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/multi_capture + sink_uri='file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-21141?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/multi_capture ++ date + echo '[Fri Apr 26 19:18:00 CST 2024] <<<<<< START storage consumer in multi_capture case >>>>>>' [Fri Apr 26 19:18:00 CST 2024] <<<<<< START storage consumer in multi_capture case >>>>>> + cd /tmp/tidb_cdc_test/multi_capture + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/multi_capture + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/multi_capture/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/multi_capture/storage_test/ticdc-multi-capture-test-21141?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table multi_capture_1.usertable not exists for 1-th check, retry later /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7943) - No such process wait process 7943 exit for 3-th time... process 7943 already exit [Fri Apr 26 19:18:01 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.81198121.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 31-th check, retry later check diff failed 5-th time, retry later table processor_delay.t48 not exists for 3-th check, retry later table multi_capture_1.usertable not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 32-th check, retry later check diff failed 6-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:04 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133359178103920 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:17:45.83662699 +0800 CST StartTs:449348964372971529 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031a8fc0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348964399185925} {CheckpointTs:449348967872593922 MinTableBarrierTs:449348969183313923 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19acd425-fa7c-466e-a935-6a5ea01d2314 {"id":"19acd425-fa7c-466e-a935-6a5ea01d2314","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdacd92 19acd425-fa7c-466e-a935-6a5ea01d2314 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133359178103920,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:17:45.83662699+08:00","start-ts":449348964372971529,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348964399185925} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348967872593922,"min-table-barrier-ts":449348969183313923,"admin-job-type":0} /tidb/cdc/default/default/task/position/19acd425-fa7c-466e-a935-6a5ea01d2314/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133359178103920 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:17:45.83662699 +0800 CST StartTs:449348964372971529 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031a8fc0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348964399185925} {CheckpointTs:449348967872593922 MinTableBarrierTs:449348969183313923 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19acd425-fa7c-466e-a935-6a5ea01d2314 {"id":"19acd425-fa7c-466e-a935-6a5ea01d2314","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdacd92 19acd425-fa7c-466e-a935-6a5ea01d2314 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133359178103920,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:17:45.83662699+08:00","start-ts":449348964372971529,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348964399185925} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348967872593922,"min-table-barrier-ts":449348969183313923,"admin-job-type":0} /tidb/cdc/default/default/task/position/19acd425-fa7c-466e-a935-6a5ea01d2314/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133359178103920 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:17:45.83662699 +0800 CST StartTs:449348964372971529 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031a8fc0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348964399185925} {CheckpointTs:449348967872593922 MinTableBarrierTs:449348969183313923 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348967872593922, checkpointTs: 449348967872593922, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/19acd425-fa7c-466e-a935-6a5ea01d2314 {"id":"19acd425-fa7c-466e-a935-6a5ea01d2314","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdacd92 19acd425-fa7c-466e-a935-6a5ea01d2314 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133359178103920,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:17:45.83662699+08:00","start-ts":449348964372971529,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348964399185925} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348967872593922,"min-table-barrier-ts":449348969183313923,"admin-job-type":0} /tidb/cdc/default/default/task/position/19acd425-fa7c-466e-a935-6a5ea01d2314/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table processor_delay.t48 not exists for 4-th check, retry later table multi_capture_1.usertable not exists for 3-th check, retry later check_ts_not_forward ddl-only-block-related-table table ddl_manager.finish_mark not exists for 33-th check, retry later check diff failed 7-th time, retry later table multi_capture_1.usertable not exists for 4-th check, retry later table processor_delay.t48 not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 34-th check, retry later table multi_capture_1.usertable not exists for 5-th check, retry later check diff failed 8-th time, retry later ls /tmp/tidb_cdc_test/storage_csv_update/storage_test/test/test_update 449348961997422625 run task successfully start check update + workdir=/tmp/tidb_cdc_test/storage_csv_update + sink_uri='file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/storage_csv_update ++ date + echo '[Fri Apr 26 19:18:07 CST 2024] <<<<<< START storage consumer in storage_csv_update case >>>>>>' [Fri Apr 26 19:18:07 CST 2024] <<<<<< START storage consumer in storage_csv_update case >>>>>> + cd /tmp/tidb_cdc_test/storage_csv_update + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/storage_csv_update + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/storage_csv_update/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/storage_csv_update/storage_test?flush-interval=5s&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/storage_csv_update/conf/changefeed.toml table processor_delay.t48 exists table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 35-th check, retry later check diff failed 9-th time, retry later table processor_delay.t50 not exists for 2-th check, retry later table multi_capture_1.usertable not exists for 6-th check, retry later table ddl_manager.finish_mark not exists for 36-th check, retry later check diff successfully table multi_capture_1.usertable exists table multi_capture_2.usertable exists table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:18:12 CST 2024] <<<<<< run test case generate_column success! >>>>>> table processor_delay.t50 not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 37-th check, retry later check diff failed 1-th time, retry later table multi_capture_4.usertable not exists for 2-th check, retry later table processor_delay.t50 not exists for 4-th check, retry later check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 38-th check, retry later check diff failed 2-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } table processor_delay.t50 not exists for 5-th check, retry later Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) check diff failed 2-th time, retry later [Pipeline] // cache [Pipeline] } table multi_capture_4.usertable not exists for 3-th check, retry later [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } table ddl_manager.finish_mark not exists for 39-th check, retry later [Pipeline] // stage [Pipeline] } check diff failed 3-th time, retry later table processor_delay.t50 exists check diff successfully [Fri Apr 26 19:18:19 CST 2024] <<<<<< run test case storage_csv_update success! >>>>>> table multi_capture_4.usertable not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 40-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... table multi_capture_4.usertable not exists for 5-th check, retry later check diff failed 4-th time, retry later wait process cdc.test exit for 2-th time... table ddl_manager.finish_mark not exists for 41-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:18:22 CST 2024] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> table multi_capture_4.usertable exists check diff failed 5-th time, retry later check diff successfully ***************** properties ***************** "scanproportion"="0" "mysql.user"="root" "readproportion"="0" "updateproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" "insertproportion"="0" "requestdistribution"="uniform" "mysql.port"="4000" "threadcount"="2" "workload"="core" "dotransactions"="false" "readallfields"="true" "mysql.db"="multi_capture_1" "recordcount"="20" ********************************************** Run finished, takes 68.50576ms INSERT - Takes(s): 0.1, Count: 20, OPS: 391.2, Avg(us): 6082, Min(us): 1744, Max(us): 17242, 95th(us): 17000, 99th(us): 18000 ***************** properties ***************** "scanproportion"="0" "workload"="core" "updateproportion"="0" "mysql.user"="root" "insertproportion"="0" "mysql.host"="127.0.0.1" "mysql.db"="multi_capture_2" "dotransactions"="false" "recordcount"="20" "requestdistribution"="uniform" "operationcount"="0" "mysql.port"="4000" "readproportion"="0" "threadcount"="2" "readallfields"="true" ********************************************** Run finished, takes 59.618191ms INSERT - Takes(s): 0.1, Count: 20, OPS: 374.2, Avg(us): 4973, Min(us): 1998, Max(us): 12114, 95th(us): 8000, 99th(us): 13000 ***************** properties ***************** "readproportion"="0" "scanproportion"="0" "mysql.db"="multi_capture_3" "threadcount"="2" "dotransactions"="false" "recordcount"="20" "operationcount"="0" "mysql.host"="127.0.0.1" "updateproportion"="0" "readallfields"="true" "requestdistribution"="uniform" "mysql.user"="root" "mysql.port"="4000" "insertproportion"="0" "workload"="core" ********************************************** Run finished, takes 53.484932ms INSERT - Takes(s): 0.0, Count: 20, OPS: 410.8, Avg(us): 5015, Min(us): 1877, Max(us): 11654, 95th(us): 10000, 99th(us): 12000 ***************** properties ***************** "readallfields"="true" "insertproportion"="0" "dotransactions"="false" "mysql.user"="root" "requestdistribution"="uniform" "threadcount"="2" "updateproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "recordcount"="20" "mysql.db"="multi_capture_4" "scanproportion"="0" "mysql.port"="4000" "operationcount"="0" "readproportion"="0" ********************************************** Run finished, takes 50.096304ms INSERT - Takes(s): 0.0, Count: 20, OPS: 465.7, Avg(us): 4627, Min(us): 1405, Max(us): 8567, 95th(us): 8000, 99th(us): 9000 table ddl_manager.finish_mark not exists for 42-th check, retry later check diff failed 1-th time, retry later check diff failed 6-th time, retry later table ddl_manager.finish_mark not exists for 43-th check, retry later check diff failed 2-th time, retry later check diff failed 7-th time, retry later table ddl_manager.finish_mark not exists for 44-th check, retry later check diff failed 3-th time, retry later check diff failed 8-th time, retry later table ddl_manager.finish_mark not exists for 45-th check, retry later check diff failed 4-th time, retry later check diff successfully table ddl_manager.finish_mark not exists for 46-th check, retry later check diff failed 1-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 47-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } wait process cdc.test exit for 2-th time... Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv wait process cdc.test exit for 3-th time... [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } check diff failed 2-th time, retry later wait process cdc.test exit for 4-th time... table ddl_manager.finish_mark not exists for 48-th check, retry later check diff failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 5-th time... process cdc.test already exit [Fri Apr 26 19:18:37 CST 2024] <<<<<< run test case multi_capture success! >>>>>> table ddl_manager.finish_mark not exists for 49-th check, retry later check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 50-th check, retry later check diff failed 5-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table ddl_manager.finish_mark not exists for 51-th check, retry later check diff failed 6-th time, retry later table ddl_manager.finish_mark not exists for 52-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 7-th time, retry later table ddl_manager.finish_mark not exists for 53-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 8-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 54-th check, retry later check diff failed 9-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68848ab80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:8138, start at 2024-04-26 19:18:50.315090992 +0800 CST m=+5.133507390 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:50.322 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:50.286 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:50.286 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68848ab80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:8138, start at 2024-04-26 19:18:50.315090992 +0800 CST m=+5.133507390 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:50.322 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:50.286 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:50.286 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68848c300013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-5ssms, pid:8226, start at 2024-04-26 19:18:50.415182287 +0800 CST m=+5.170133401 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:50.423 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:50.380 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:50.380 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table ddl_manager.finish_mark not exists for 55-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully [Fri Apr 26 19:18:53 CST 2024] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.94869488.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 56-th check, retry later check diff failed 2-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce56cc59-629b-44f9-bcbf-eaba62f563fd {"id":"ce56cc59-629b-44f9-bcbf-eaba62f563fd","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20fcc4f1 ce56cc59-629b-44f9-bcbf-eaba62f563fd /tidb/cdc/default/default/upstream/7362133678152654336 {"id":7362133678152654336,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce56cc59-629b-44f9-bcbf-eaba62f563fd {"id":"ce56cc59-629b-44f9-bcbf-eaba62f563fd","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20fcc4f1 ce56cc59-629b-44f9-bcbf-eaba62f563fd /tidb/cdc/default/default/upstream/7362133678152654336 {"id":7362133678152654336,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce56cc59-629b-44f9-bcbf-eaba62f563fd {"id":"ce56cc59-629b-44f9-bcbf-eaba62f563fd","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20fcc4f1 ce56cc59-629b-44f9-bcbf-eaba62f563fd /tidb/cdc/default/default/upstream/7362133678152654336 {"id":7362133678152654336,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/sink_hang + sink_uri='file:///tmp/tidb_cdc_test/sink_hang/storage_test/ticdc-sink-hang-test-23745?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/sink_hang ++ date + echo '[Fri Apr 26 19:18:56 CST 2024] <<<<<< START storage consumer in sink_hang case >>>>>>' [Fri Apr 26 19:18:56 CST 2024] <<<<<< START storage consumer in sink_hang case >>>>>> + cd /tmp/tidb_cdc_test/sink_hang + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/sink_hang + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/sink_hang/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/sink_hang/storage_test/ticdc-sink-hang-test-23745?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table ddl_manager.finish_mark not exists for 57-th check, retry later check diff failed 3-th time, retry later check_changefeed_status 127.0.0.1:8300 44c08afa-9dd6-4ac0-bcf2-fe297340c05f normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=44c08afa-9dd6-4ac0-bcf2-fe297340c05f + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/44c08afa-9dd6-4ac0-bcf2-fe297340c05f/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1082 0 --:--:-- --:--:-- --:--:-- 1088 + info='{"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730}' + echo '{"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730}' {"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730} ++ echo '{"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 44c08afa-9dd6-4ac0-bcf2-fe297340c05f normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=44c08afa-9dd6-4ac0-bcf2-fe297340c05f + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/44c08afa-9dd6-4ac0-bcf2-fe297340c05f/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 1075 0 --:--:-- --:--:-- --:--:-- 1088 + info='{"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730}' + echo '{"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730}' {"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730} ++ echo '{"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":449348983163977730,"checkpoint_ts":449348983163977730}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 58-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } check diff failed 4-th time, retry later [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table sink_hang.t1 not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 59-th check, retry later check diff failed 5-th time, retry later table sink_hang.t1 not exists for 3-th check, retry later table ddl_manager.finish_mark not exists for 60-th check, retry later check diff failed 6-th time, retry later table sink_hang.t1 not exists for 4-th check, retry later table ddl_manager.finish_mark not exists for 61-th check, retry later check diff failed 7-th time, retry later table sink_hang.t1 not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 62-th check, retry later check diff failed 8-th time, retry later table sink_hang.t1 exists table sink_hang.t2 exists check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 63-th check, retry later table big_txn.finish_mark_1 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:19:01 CST 2024] <<<<<< run test case big_txn success! >>>>>> check diff failed 9-th time, retry later check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 64-th check, retry later pass check, checkpoint tso not forward after 10s run task successfully wait process 8124 exit for 1-th time... wait process 8124 exit for 2-th time... wait process 8124 exit for 3-th time... wait process 8124 exit for 4-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8124) - No such process wait process 8124 exit for 5-th time... process 8124 already exit [Fri Apr 26 19:19:08 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.86358637.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:11 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133359178103920 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:17:45.83662699 +0800 CST StartTs:449348964372971529 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00240ccf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348964399185925} {CheckpointTs:449348969353969678 MinTableBarrierTs:449348986760069124 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/92ea6431-434f-463c-8dbf-9f6b6d977785 {"id":"92ea6431-434f-463c-8dbf-9f6b6d977785","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdace78 92ea6431-434f-463c-8dbf-9f6b6d977785 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133359178103920,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:17:45.83662699+08:00","start-ts":449348964372971529,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348964399185925} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348969353969678,"min-table-barrier-ts":449348986760069124,"admin-job-type":0} /tidb/cdc/default/default/task/position/92ea6431-434f-463c-8dbf-9f6b6d977785/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133359178103920 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:17:45.83662699 +0800 CST StartTs:449348964372971529 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00240ccf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348964399185925} {CheckpointTs:449348969353969678 MinTableBarrierTs:449348986760069124 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/92ea6431-434f-463c-8dbf-9f6b6d977785 {"id":"92ea6431-434f-463c-8dbf-9f6b6d977785","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdace78 92ea6431-434f-463c-8dbf-9f6b6d977785 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133359178103920,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:17:45.83662699+08:00","start-ts":449348964372971529,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348964399185925} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348969353969678,"min-table-barrier-ts":449348986760069124,"admin-job-type":0} /tidb/cdc/default/default/task/position/92ea6431-434f-463c-8dbf-9f6b6d977785/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133359178103920 Namespace:default ID:ddl-only-block-related-table SinkURI:file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json&enable-tidb-extension=true CreateTime:2024-04-26 19:17:45.83662699 +0800 CST StartTs:449348964372971529 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00240ccf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g4a2af7498 Epoch:449348964399185925} {CheckpointTs:449348969353969678 MinTableBarrierTs:449348986760069124 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348969353969678, checkpointTs: 449348969353969678, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/92ea6431-434f-463c-8dbf-9f6b6d977785 {"id":"92ea6431-434f-463c-8dbf-9f6b6d977785","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdace78 92ea6431-434f-463c-8dbf-9f6b6d977785 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133359178103920,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"file:///tmp/tidb_cdc_test/ddl_only_block_related_table/storage_test/ticdc-common-1-test-13117?protocol=canal-json\u0026enable-tidb-extension=true","create-time":"2024-04-26T19:17:45.83662699+08:00","start-ts":449348964372971529,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"file-index-digit":20,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g4a2af7498","epoch":449348964399185925} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348969353969678,"min-table-barrier-ts":449348986760069124,"admin-job-type":0} /tidb/cdc/default/default/task/position/92ea6431-434f-463c-8dbf-9f6b6d977785/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133359178103920 {"id":7362133359178103920,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff successfully check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 65-th check, retry later check diff failed 2-th time, retry later check diff failed 1-th time, retry later check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 66-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check diff failed 3-th time, retry later check diff failed 2-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 67-th check, retry later wait process cdc.test exit for 2-th time... check diff successfully check_ts_forward ddl-only-block-related-table cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:18 CST 2024] <<<<<< run test case sink_hang success! >>>>>> check diff failed 3-th time, retry later table ddl_manager.finish_mark not exists for 68-th check, retry later changefeed is working normally rts: 449348988595077100->449348988857221100 checkpoint: 449348988595077100->449348988857221100 run task successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 1-th time... check diff failed 4-th time, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:21 CST 2024] <<<<<< run test case ddl_only_block_related_table success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... table ddl_manager.finish_mark not exists for 69-th check, retry later Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 70-th check, retry later check diff failed 6-th time, retry later table ddl_manager.finish_mark exists ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully wait process cdc.test exit for 1-th time... check diff failed 7-th time, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886c894000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:3972, start at 2024-04-26 19:19:27.026883009 +0800 CST m=+5.225501158 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:27.033 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:27.013 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:27.013 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:19:28 CST 2024] <<<<<< run test case ddl_manager success! >>>>>> check diff failed 8-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886c894000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:3972, start at 2024-04-26 19:19:27.026883009 +0800 CST m=+5.225501158 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:27.033 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:27.013 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:27.013 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886ca0c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:4055, start at 2024-04-26 19:19:27.156830019 +0800 CST m=+5.292744259 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:27.164 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:27.157 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:27.157 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } check diff successfully [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Fri Apr 26 19:19:32 CST 2024] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.54265428.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:35 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27 {"id":"b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2185faf0 b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27 /tidb/cdc/default/default/upstream/7362133832807091170 {"id":7362133832807091170,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27 {"id":"b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2185faf0 b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27 /tidb/cdc/default/default/upstream/7362133832807091170 {"id":7362133832807091170,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27 {"id":"b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2185faf0 b929c3a0-fc0c-4fa3-8e8c-0e428ef20e27 /tidb/cdc/default/default/upstream/7362133832807091170 {"id":7362133832807091170,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/changefeed_finish + sink_uri='file:///tmp/tidb_cdc_test/changefeed_finish/storage_test/ticdc-changefeed-pause-resume-32480?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_finish ++ date + echo '[Fri Apr 26 19:19:35 CST 2024] <<<<<< START storage consumer in changefeed_finish case >>>>>>' [Fri Apr 26 19:19:35 CST 2024] <<<<<< START storage consumer in changefeed_finish case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_finish + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_finish + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_finish/storage_test/ticdc-changefeed-pause-resume-32480?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check diff failed 2-th time, retry later check diff failed 1-th time, retry later check diff failed 3-th time, retry later check diff failed 2-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... check diff failed 4-th time, retry later check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check diff failed 5-th time, retry later check diff failed 4-th time, retry later Verifying downstream PD is started... check diff failed 6-th time, retry later check diff failed 5-th time, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff failed 7-th time, retry later check diff successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: storage... <<================= [Fri Apr 26 19:19:44 CST 2024] <<<<<< run test case multi_source success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 8-th time, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } check diff failed 9-th time, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68884bcc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:9804, start at 2024-04-26 19:19:51.813950193 +0800 CST m=+5.032449848 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:51.821 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:51.795 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:51.795 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68884bcc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:9804, start at 2024-04-26 19:19:51.813950193 +0800 CST m=+5.032449848 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:51.821 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:51.795 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:51.795 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68884c7c0008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-16m8q, pid:9881, start at 2024-04-26 19:19:51.846829783 +0800 CST m=+5.001337702 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:51.852 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:51.839 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:51.839 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:53 CST 2024] <<<<<< run test case changefeed_pause_resume success! >>>>>> ***************** properties ***************** "mysql.host"="127.0.0.1" "updateproportion"="0" "requestdistribution"="uniform" "recordcount"="20" "operationcount"="0" "readproportion"="0" "threadcount"="4" "workload"="core" "insertproportion"="0" "mysql.user"="root" "scanproportion"="0" "mysql.db"="changefeed_auto_stop_1" "mysql.port"="4000" "dotransactions"="false" "readallfields"="true" ********************************************** Run finished, takes 10.57935ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3491.5, Avg(us): 1923, Min(us): 1006, Max(us): 4771, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "readallfields"="true" "mysql.host"="127.0.0.1" "scanproportion"="0" "insertproportion"="0" "requestdistribution"="uniform" "updateproportion"="0" "threadcount"="4" "mysql.user"="root" "dotransactions"="false" "operationcount"="0" "readproportion"="0" "mysql.port"="4000" "mysql.db"="changefeed_auto_stop_2" "workload"="core" "recordcount"="20" ********************************************** Run finished, takes 10.010189ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3491.0, Avg(us): 1824, Min(us): 860, Max(us): 4965, 95th(us): 5000, 99th(us): 5000 ***************** properties ***************** "readallfields"="true" "threadcount"="4" "mysql.port"="4000" "mysql.db"="changefeed_auto_stop_3" "readproportion"="0" "operationcount"="0" "scanproportion"="0" "recordcount"="20" "updateproportion"="0" "requestdistribution"="uniform" "mysql.user"="root" "workload"="core" "insertproportion"="0" "dotransactions"="false" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 8.455476ms INSERT - Takes(s): 0.0, Count: 20, OPS: 4440.7, Avg(us): 1605, Min(us): 855, Max(us): 3993, 95th(us): 4000, 99th(us): 4000 ***************** properties ***************** "operationcount"="0" "mysql.user"="root" "workload"="core" "mysql.port"="4000" "requestdistribution"="uniform" "mysql.db"="changefeed_auto_stop_4" "insertproportion"="0" "recordcount"="20" "readallfields"="true" "scanproportion"="0" "mysql.host"="127.0.0.1" "readproportion"="0" "dotransactions"="false" "updateproportion"="0" "threadcount"="4" ********************************************** Run finished, takes 9.51002ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3698.6, Avg(us): 1809, Min(us): 990, Max(us): 4121, 95th(us): 5000, 99th(us): 5000 [Fri Apr 26 19:19:56 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1138211384.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:59 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 {"id":"0bec7f4b-dfa6-49fd-8e2e-e31f1e622140","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb493c 0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 /tidb/cdc/default/default/upstream/7362133933641843994 {"id":7362133933641843994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 {"id":"0bec7f4b-dfa6-49fd-8e2e-e31f1e622140","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb493c 0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 /tidb/cdc/default/default/upstream/7362133933641843994 {"id":7362133933641843994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 {"id":"0bec7f4b-dfa6-49fd-8e2e-e31f1e622140","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb493c 0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 /tidb/cdc/default/default/upstream/7362133933641843994 {"id":7362133933641843994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:19:59 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.1142711429.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:02 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 {"id":"0bec7f4b-dfa6-49fd-8e2e-e31f1e622140","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/d6ed85b3-07d4-4d15-8d65-086a847d3b06 {"id":"d6ed85b3-07d4-4d15-8d65-086a847d3b06","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb493c 0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb4969 d6ed85b3-07d4-4d15-8d65-086a847d3b06 /tidb/cdc/default/default/upstream/7362133933641843994 {"id":7362133933641843994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 {"id":"0bec7f4b-dfa6-49fd-8e2e-e31f1e622140","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/d6ed85b3-07d4-4d15-8d65-086a847d3b06 {"id":"d6ed85b3-07d4-4d15-8d65-086a847d3b06","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb493c 0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb4969 d6ed85b3-07d4-4d15-8d65-086a847d3b06 /tidb/cdc/default/default/upstream/7362133933641843994 {"id":7362133933641843994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 {"id":"0bec7f4b-dfa6-49fd-8e2e-e31f1e622140","address":"127.0.0.1:8301","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/capture/d6ed85b3-07d4-4d15-8d65-086a847d3b06 {"id":"d6ed85b3-07d4-4d15-8d65-086a847d3b06","address":"127.0.0.1:8302","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb493c 0bec7f4b-dfa6-49fd-8e2e-e31f1e622140 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21eb4969 d6ed85b3-07d4-4d15-8d65-086a847d3b06 /tidb/cdc/default/default/upstream/7362133933641843994 {"id":7362133933641843994,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + workdir=/tmp/tidb_cdc_test/changefeed_auto_stop + sink_uri='file:///tmp/tidb_cdc_test/changefeed_auto_stop/storage_test/ticdc-changefeed-auto-stop-test-18520?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/changefeed_auto_stop ++ date + echo '[Fri Apr 26 19:20:02 CST 2024] <<<<<< START storage consumer in changefeed_auto_stop case >>>>>>' [Fri Apr 26 19:20:02 CST 2024] <<<<<< START storage consumer in changefeed_auto_stop case >>>>>> + cd /tmp/tidb_cdc_test/changefeed_auto_stop + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/changefeed_auto_stop + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/changefeed_auto_stop/storage_test/ticdc-changefeed-auto-stop-test-18520?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' check_changefeed_state http://127.0.0.1:2379 52aa95fc-5e1e-4277-bb7b-c545e222f331 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=52aa95fc-5e1e-4277-bb7b-c545e222f331 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 52aa95fc-5e1e-4277-bb7b-c545e222f331 -s + info='{ "upstream_id": 7362133933641843994, "namespace": "default", "id": "52aa95fc-5e1e-4277-bb7b-c545e222f331", "state": "normal", "checkpoint_tso": 449348998318260225, "checkpoint_time": "2024-04-26 19:19:55.196", "error": null }' + echo '{ "upstream_id": 7362133933641843994, "namespace": "default", "id": "52aa95fc-5e1e-4277-bb7b-c545e222f331", "state": "normal", "checkpoint_tso": 449348998318260225, "checkpoint_time": "2024-04-26 19:19:55.196", "error": null }' { "upstream_id": 7362133933641843994, "namespace": "default", "id": "52aa95fc-5e1e-4277-bb7b-c545e222f331", "state": "normal", "checkpoint_tso": 449348998318260225, "checkpoint_time": "2024-04-26 19:19:55.196", "error": null } ++ echo '{' '"upstream_id":' 7362133933641843994, '"namespace":' '"default",' '"id":' '"52aa95fc-5e1e-4277-bb7b-c545e222f331",' '"state":' '"normal",' '"checkpoint_tso":' 449348998318260225, '"checkpoint_time":' '"2024-04-26' '19:19:55.196",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362133933641843994, '"namespace":' '"default",' '"id":' '"52aa95fc-5e1e-4277-bb7b-c545e222f331",' '"state":' '"normal",' '"checkpoint_tso":' 449348998318260225, '"checkpoint_time":' '"2024-04-26' '19:19:55.196",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later table changefeed_auto_stop_1.usertable not exists for 2-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/cli/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... table changefeed_auto_stop_1.usertable not exists for 3-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/cli Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table changefeed_auto_stop_1.usertable not exists for 4-th check, retry later table changefeed_auto_stop_1.usertable not exists for 5-th check, retry later table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_auto_stop_3.usertable not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_auto_stop_3.usertable not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_auto_stop_3.usertable not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688a0f140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:11584, start at 2024-04-26 19:20:20.71444885 +0800 CST m=+5.326242391 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:20.724 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:20.728 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:20.728 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688a0f140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:11584, start at 2024-04-26 19:20:20.71444885 +0800 CST m=+5.326242391 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:20.724 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:20.728 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:20.728 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688a0e9c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:11655, start at 2024-04-26 19:20:20.692118937 +0800 CST m=+5.237088046 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:20.699 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:20.696 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:20.696 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cli/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cli/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.12937.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 2-th time, retry later + set +x + tso='449349006387576833 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349006387576833 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x check diff failed 3-th time, retry later [Fri Apr 26 19:20:28 CST 2024] <<<<<< START cdc server in cli case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.1297612978.out server --log-file /tmp/tidb_cdc_test/cli/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:31 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e7636ea-c3b3-48a2-9116-f4a9b5c184c4 {"id":"8e7636ea-c3b3-48a2-9116-f4a9b5c184c4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a225b080a 8e7636ea-c3b3-48a2-9116-f4a9b5c184c4 /tidb/cdc/default/default/upstream/7362134060816647452 {"id":7362134060816647452,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e7636ea-c3b3-48a2-9116-f4a9b5c184c4 {"id":"8e7636ea-c3b3-48a2-9116-f4a9b5c184c4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a225b080a 8e7636ea-c3b3-48a2-9116-f4a9b5c184c4 /tidb/cdc/default/default/upstream/7362134060816647452 {"id":7362134060816647452,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e7636ea-c3b3-48a2-9116-f4a9b5c184c4 {"id":"8e7636ea-c3b3-48a2-9116-f4a9b5c184c4","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a225b080a 8e7636ea-c3b3-48a2-9116-f4a9b5c184c4 /tidb/cdc/default/default/upstream/7362134060816647452 {"id":7362134060816647452,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13025.out cli changefeed create --start-ts=449349006387576833 '--sink-uri=file:///tmp/tidb_cdc_test/cli/storage_test/ticdc-cli-test-17987?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7362134060816647452,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli/storage_test/ticdc-cli-test-17987?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:20:31.986647591+08:00","start_ts":449349006387576833,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349006387576833,"checkpoint_ts":449349006387576833,"checkpoint_time":"2024-04-26 19:20:25.978"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check diff failed 5-th time, retry later + set +x + workdir=/tmp/tidb_cdc_test/cli + sink_uri='file:///tmp/tidb_cdc_test/cli/storage_test/ticdc-cli-test-17987?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/cli ++ date + echo '[Fri Apr 26 19:20:33 CST 2024] <<<<<< START storage consumer in cli case >>>>>>' [Fri Apr 26 19:20:33 CST 2024] <<<<<< START storage consumer in cli case >>>>>> + cd /tmp/tidb_cdc_test/cli + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/cli + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/cli/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/cli/storage_test/ticdc-cli-test-17987?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table test.simple not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table test.simple not exists for 2-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:20:36 CST 2024] <<<<<< run test case changefeed_auto_stop success! >>>>>> table test.simple not exists for 3-th check, retry later table test.simple not exists for 4-th check, retry later table test.simple not exists for 5-th check, retry later table test.simple not exists for 6-th check, retry later table test.simple exists table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349011237240835, "checkpoint_time": "2024-04-26 19:20:44.478", "error": null }' + echo '{ "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349011237240835, "checkpoint_time": "2024-04-26 19:20:44.478", "error": null }' { "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349011237240835, "checkpoint_time": "2024-04-26 19:20:44.478", "error": null } ++ echo '{' '"upstream_id":' 7362134060816647452, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449349011237240835, '"checkpoint_time":' '"2024-04-26' '19:20:44.478",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362134060816647452, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449349011237240835, '"checkpoint_time":' '"2024-04-26' '19:20:44.478",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc002d91078} {Type:update Path:[Config SyncPointRetention] From: To:0xc002d91090} {Type:update Path:[Config Consistent] From: To:0xc0011d4a80} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13366.out cli changefeed --changefeed-id custom-changefeed-name pause PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } + set +x [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449349011761528835, "checkpoint_time": "2024-04-26 19:20:46.478", "error": null }' + echo '{ "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449349011761528835, "checkpoint_time": "2024-04-26 19:20:46.478", "error": null }' { "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449349011761528835, "checkpoint_time": "2024-04-26 19:20:46.478", "error": null } ++ echo '{' '"upstream_id":' 7362134060816647452, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449349011761528835, '"checkpoint_time":' '"2024-04-26' '19:20:46.478",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7362134060816647452, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449349011761528835, '"checkpoint_time":' '"2024-04-26' '19:20:46.478",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13438.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config SyncPointInterval] From: To:0xc002fa8108} {Type:update Path:[Config SyncPointRetention] From: To:0xc002fa8118} {Type:update Path:[Config Consistent] From: To:0xc000a49340} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7362134060816647452,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli/storage_test/ticdc-cli-test-17987?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:20:31.986647591+08:00","start_ts":449349006387576833,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"delete_only_output_handle_key_columns":null,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349011761528836,"checkpoint_ts":449349011761528835,"checkpoint_time":"2024-04-26 19:20:46.478"} PASS coverage: 2.9% of statements in github.com/pingcap/tiflow/... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13476.out cli changefeed --changefeed-id custom-changefeed-name resume PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... + set +x + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349014907256836, "checkpoint_time": "2024-04-26 19:20:58.478", "error": null }' + echo '{ "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349014907256836, "checkpoint_time": "2024-04-26 19:20:58.478", "error": null }' { "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349014907256836, "checkpoint_time": "2024-04-26 19:20:58.478", "error": null } ++ echo '{' '"upstream_id":' 7362134060816647452, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449349014907256836, '"checkpoint_time":' '"2024-04-26' '19:20:58.478",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362134060816647452, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449349014907256836, '"checkpoint_time":' '"2024-04-26' '19:20:58.478",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13554.out cli changefeed --changefeed-id custom-changefeed-name remove Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 449349015169400836 SinkURI: file:///tmp/tidb_cdc_test/cli/storage_test/ticdc-cli-test-17987?protocol=canal-json&enable-tidb-extension=true PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... + set +x changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13615.out cli changefeed create '--sink-uri=file:///tmp/tidb_cdc_test/cli/storage_test/ticdc-cli-test-17987?protocol=canal-json&enable-tidb-extension=true' --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7362134060816647452,"namespace":"default","id":"custom-changefeed-name","sink_uri":"file:///tmp/tidb_cdc_test/cli/storage_test/ticdc-cli-test-17987?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:21:05.136299283+08:00","start_ts":449349016624300036,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349016624300036,"checkpoint_ts":449349016624300036,"checkpoint_time":"2024-04-26 19:21:05.028"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349017528696837, "checkpoint_time": "2024-04-26 19:21:08.478", "error": null }' + echo '{ "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349017528696837, "checkpoint_time": "2024-04-26 19:21:08.478", "error": null }' { "upstream_id": 7362134060816647452, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449349017528696837, "checkpoint_time": "2024-04-26 19:21:08.478", "error": null } ++ echo '{' '"upstream_id":' 7362134060816647452, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449349017528696837, '"checkpoint_time":' '"2024-04-26' '19:21:08.478",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362134060816647452, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449349017528696837, '"checkpoint_time":' '"2024-04-26' '19:21:08.478",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13709.out cli unsafe delete-service-gc-safepoint Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13736.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 reset and all metadata truncated in PD! PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13835.out cli unsafe resolve-lock --region=34 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.13859.out cli unsafe resolve-lock --region=34 --ts=449349018957381636 PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 145k 4204 --:--:-- --:--:-- --:--:-- 207k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" check_changefeed_state http://127.0.0.1:2379 cacf7f5e-0b3d-479f-bf2c-41de434b89ef finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=cacf7f5e-0b3d-479f-bf2c-41de434b89ef + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c cacf7f5e-0b3d-479f-bf2c-41de434b89ef -s + info='{ "upstream_id": 7362133832807091170, "namespace": "default", "id": "cacf7f5e-0b3d-479f-bf2c-41de434b89ef", "state": "finished", "checkpoint_tso": 449349016698748930, "checkpoint_time": "2024-04-26 19:21:05.312", "error": null }' + echo '{ "upstream_id": 7362133832807091170, "namespace": "default", "id": "cacf7f5e-0b3d-479f-bf2c-41de434b89ef", "state": "finished", "checkpoint_tso": 449349016698748930, "checkpoint_time": "2024-04-26 19:21:05.312", "error": null }' { "upstream_id": 7362133832807091170, "namespace": "default", "id": "cacf7f5e-0b3d-479f-bf2c-41de434b89ef", "state": "finished", "checkpoint_tso": 449349016698748930, "checkpoint_time": "2024-04-26 19:21:05.312", "error": null } ++ echo '{' '"upstream_id":' 7362133832807091170, '"namespace":' '"default",' '"id":' '"cacf7f5e-0b3d-479f-bf2c-41de434b89ef",' '"state":' '"finished",' '"checkpoint_tso":' 449349016698748930, '"checkpoint_time":' '"2024-04-26' '19:21:05.312",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7362133832807091170, '"namespace":' '"default",' '"id":' '"cacf7f5e-0b3d-479f-bf2c-41de434b89ef",' '"state":' '"finished",' '"checkpoint_tso":' 449349016698748930, '"checkpoint_time":' '"2024-04-26' '19:21:05.312",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:21:18 CST 2024] <<<<<< run test case changefeed_finish success! >>>>>> } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 176 100 176 0 0 159k 0 --:--:-- --:--:-- --:--:-- 171k { "version": "v7.5.1-57-g4a2af7498", "git_hash": "4a2af7498eddb73ddf965dcfc461666a0186675b", "id": "13a8e234-4e7d-4aff-8b6e-2425d415c35c", "pid": 12981, "is_owner": true }wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:21:24 CST 2024] <<<<<< run test case cli success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: storage... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688f28e00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:6871, start at 2024-04-26 19:21:44.289965659 +0800 CST m=+5.548539514 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:44.299 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:44.298 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:44.298 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688f28e00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:6871, start at 2024-04-26 19:21:44.289965659 +0800 CST m=+5.548539514 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:44.299 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:44.298 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:44.298 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688f28b80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-zsf75, pid:6952, start at 2024-04-26 19:21:44.262632009 +0800 CST m=+5.471682154 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:44.271 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:44.238 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:44.238 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 26 19:21:49 CST 2024] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.83038305.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:21:52 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41f9adaa-b486-4600-8893-4e490f4a483d {"id":"41f9adaa-b486-4600-8893-4e490f4a483d","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a239ed3f1 41f9adaa-b486-4600-8893-4e490f4a483d /tidb/cdc/default/default/upstream/7362134417412288207 {"id":7362134417412288207,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41f9adaa-b486-4600-8893-4e490f4a483d {"id":"41f9adaa-b486-4600-8893-4e490f4a483d","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a239ed3f1 41f9adaa-b486-4600-8893-4e490f4a483d /tidb/cdc/default/default/upstream/7362134417412288207 {"id":7362134417412288207,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/41f9adaa-b486-4600-8893-4e490f4a483d {"id":"41f9adaa-b486-4600-8893-4e490f4a483d","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a239ed3f1 41f9adaa-b486-4600-8893-4e490f4a483d /tidb/cdc/default/default/upstream/7362134417412288207 {"id":7362134417412288207,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 47d0b2f2-22e6-41d6-9578-2b9b3004c164 Info: {"upstream_id":7362134417412288207,"namespace":"default","id":"47d0b2f2-22e6-41d6-9578-2b9b3004c164","sink_uri":"file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-7955?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:21:52.478193103+08:00","start_ts":449349028190093313,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349028190093313,"checkpoint_ts":449349028190093313,"checkpoint_time":"2024-04-26 19:21:49.148"} + workdir=/tmp/tidb_cdc_test/force_replicate_table + sink_uri='file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-7955?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/force_replicate_table ++ date + echo '[Fri Apr 26 19:21:52 CST 2024] <<<<<< START storage consumer in force_replicate_table case >>>>>>' [Fri Apr 26 19:21:52 CST 2024] <<<<<< START storage consumer in force_replicate_table case >>>>>> + cd /tmp/tidb_cdc_test/force_replicate_table + '[' /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml '!=' '' ']' + cd /tmp/tidb_cdc_test/force_replicate_table + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/force_replicate_table/storage_test/ticdc-force_replicate_table-test-7955?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/force_replicate_table/conf/changefeed.toml VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688f82940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:14806, start at 2024-04-26 19:21:50.027102207 +0800 CST m=+5.290625688 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:50.035 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:50.038 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:50.038 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688f82940014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:14806, start at 2024-04-26 19:21:50.027102207 +0800 CST m=+5.290625688 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:50.035 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:50.038 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:50.038 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688f82940007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:14882, start at 2024-04-26 19:21:49.996938352 +0800 CST m=+5.210036662 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:50.004 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:49.989 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:49.989 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.16176.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349029812764673 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349029812764673 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:21:56 CST 2024] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1621316215.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:21:59 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43d1d878-04f3-4e26-88df-7617c49b6d22 {"id":"43d1d878-04f3-4e26-88df-7617c49b6d22","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23b63cf4 43d1d878-04f3-4e26-88df-7617c49b6d22 /tidb/cdc/default/default/upstream/7362134445911657962 {"id":7362134445911657962,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43d1d878-04f3-4e26-88df-7617c49b6d22 {"id":"43d1d878-04f3-4e26-88df-7617c49b6d22","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23b63cf4 43d1d878-04f3-4e26-88df-7617c49b6d22 /tidb/cdc/default/default/upstream/7362134445911657962 {"id":7362134445911657962,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43d1d878-04f3-4e26-88df-7617c49b6d22 {"id":"43d1d878-04f3-4e26-88df-7617c49b6d22","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23b63cf4 43d1d878-04f3-4e26-88df-7617c49b6d22 /tidb/cdc/default/default/upstream/7362134445911657962 {"id":7362134445911657962,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.16267.out cli changefeed create --start-ts=449349029812764673 '--sink-uri=file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-32380?protocol=canal-json&enable-tidb-extension=true' Create changefeed successfully! ID: 2e6a152d-52fc-45c1-ac1f-68fb93a01d9a Info: {"upstream_id":7362134445911657962,"namespace":"default","id":"2e6a152d-52fc-45c1-ac1f-68fb93a01d9a","sink_uri":"file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-32380?protocol=canal-json\u0026enable-tidb-extension=true","create_time":"2024-04-26T19:22:00.209462181+08:00","start_ts":449349029812764673,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"protocol":"canal-json","csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"file_index_width":20,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349029812764673,"checkpoint_ts":449349029812764673,"checkpoint_time":"2024-04-26 19:21:55.338"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table force_replicate_table.t0 not exists for 1-th check, retry later table force_replicate_table.t0 not exists for 2-th check, retry later + set +x + workdir=/tmp/tidb_cdc_test/savepoint + sink_uri='file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-32380?protocol=canal-json&enable-tidb-extension=true' + consumer_replica_config= + log_suffix= ++ pwd + pwd=/tmp/tidb_cdc_test/savepoint ++ date + echo '[Fri Apr 26 19:22:01 CST 2024] <<<<<< START storage consumer in savepoint case >>>>>>' [Fri Apr 26 19:22:01 CST 2024] <<<<<< START storage consumer in savepoint case >>>>>> + cd /tmp/tidb_cdc_test/savepoint + '[' '' '!=' '' ']' + cd /tmp/tidb_cdc_test/savepoint + set +x + cdc_storage_consumer --log-file /tmp/tidb_cdc_test/savepoint/cdc_storage_consumer.log --log-level debug --upstream-uri 'file:///tmp/tidb_cdc_test/savepoint/storage_test/ticdc-savepoint-test-32380?protocol=canal-json&enable-tidb-extension=true' --downstream-uri 'mysql://root@127.0.0.1:3306/?safe-mode=true&batch-dml-enable=false' table force_replicate_table.t0 exists table force_replicate_table.t1 exists table force_replicate_table.t2 exists table force_replicate_table.t3 exists table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later table savepoint.finish_mark not exists for 1-th check, retry later table force_replicate_table.t5 not exists for 2-th check, retry later table savepoint.finish_mark not exists for 2-th check, retry later table savepoint.finish_mark not exists for 3-th check, retry later table force_replicate_table.t5 not exists for 3-th check, retry later table savepoint.finish_mark not exists for 4-th check, retry later table force_replicate_table.t5 not exists for 4-th check, retry later table force_replicate_table.t5 not exists for 5-th check, retry later table savepoint.finish_mark not exists for 5-th check, retry later table force_replicate_table.t5 not exists for 6-th check, retry later table savepoint.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:22:14 CST 2024] <<<<<< run test case savepoint success! >>>>>> table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later table force_replicate_table.t6 not exists for 2-th check, retry later table force_replicate_table.t6 not exists for 3-th check, retry later table force_replicate_table.t6 not exists for 4-th check, retry later table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: storage... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=storage + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 2-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68927adc000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:17346, start at 2024-04-26 19:22:38.658133635 +0800 CST m=+5.267816390 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:38.666 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:38.647 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:38.647 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=7,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 4-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68927adc000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:17346, start at 2024-04-26 19:22:38.658133635 +0800 CST m=+5.267816390 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:38.666 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:38.647 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:38.647 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68927c540004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:17431, start at 2024-04-26 19:22:38.745317108 +0800 CST m=+5.290555815 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:38.752 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:38.741 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:38.741 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.18680.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349042568429569 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349042568429569 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349042568429569 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:22:45 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1871218714.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:22:48 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/20fb4702-d681-40d2-8916-b0c539c4fc7e {"id":"20fb4702-d681-40d2-8916-b0c539c4fc7e","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a247a29f4 20fb4702-d681-40d2-8916-b0c539c4fc7e /tidb/cdc/default/default/upstream/7362134653720357600 {"id":7362134653720357600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/20fb4702-d681-40d2-8916-b0c539c4fc7e {"id":"20fb4702-d681-40d2-8916-b0c539c4fc7e","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a247a29f4 20fb4702-d681-40d2-8916-b0c539c4fc7e /tidb/cdc/default/default/upstream/7362134653720357600 {"id":7362134653720357600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/20fb4702-d681-40d2-8916-b0c539c4fc7e {"id":"20fb4702-d681-40d2-8916-b0c539c4fc7e","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a247a29f4 20fb4702-d681-40d2-8916-b0c539c4fc7e /tidb/cdc/default/default/upstream/7362134653720357600 {"id":7362134653720357600,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349042568429569 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.18764.out cli changefeed create --start-ts=449349042568429569 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362134653720357600,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:22:48.903229804+08:00","start_ts":449349042568429569,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349042568429569,"checkpoint_ts":449349042568429569,"checkpoint_time":"2024-04-26 19:22:43.997"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 run task successfully wait process cdc.test exit for 1-th time... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2158 0 --:--:-- --:--:-- --:--:-- 2166 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:22:43.997","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-04-26 19:22:50.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:22:43.997","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:22:50.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:22:43.997","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:22:50.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-04-26 19:22:43.997' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:22:43.997","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:22:50.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:22:43.997","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:22:50.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-04-26 19:22:50' + echo 'sink_checkpoint_ts is 2024-04-26' 19:22:43.997 sink_checkpoint_ts is 2024-04-26 19:22:43.997 ++ date -d '2024-04-26 19:22:43.997' +%s + checkpoint_timestamp=1714130563 ++ date -d '2024-04-26 19:22:50' +%s + current_timestamp=1714130570 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:22:51 CST 2024] <<<<<< run test case force_replicate_table success! >>>>>> table test.t1 exists + sleep 5 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 1640 0 --:--:-- --:--:-- --:--:-- 1641 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:22:56.447","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:22:50.647","now_ts":"2024-04-26 19:22:57.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:22:56.447","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:22:50.647","now_ts":"2024-04-26' '19:22:57.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:22:56.447","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:22:50.647","now_ts":"2024-04-26' '19:22:57.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2644 0 --:--:-- --:--:-- --:--:-- 2662 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:25:06.597","puller_resolved_ts":"2024-04-26 19:24:59.597","last_synced_ts":"2024-04-26 19:22:50.647","now_ts":"2024-04-26 19:25:07.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:25:06.597","puller_resolved_ts":"2024-04-26' '19:24:59.597","last_synced_ts":"2024-04-26' '19:22:50.647","now_ts":"2024-04-26' '19:25:07.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 16629 7.4 0.0 13362688 140448 ? Sl 19:22 0:11 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 16685 5.3 0.0 13583308 134800 ? Sl 19:22 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1714130713.692152,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002e776c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714130713.6922333,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714130713.775775,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003392fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714130713.775843,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714130714.6221168,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00256fa40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714130714.6221986,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:25:18.460936+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f30700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:18.461205+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000dee540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:18.55182+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00098f180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:24.461884+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f30700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:24.462278+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000dee540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:24.55332+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00098f180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:25:30.462397+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f30700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:30.463352+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000dee540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:30.554036+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00098f180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:25:36.464286+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000dee540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:36.464455+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f30700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:36.55495+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00098f180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:25:38.454667+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f30700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:25:38.454719+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:25:38.455187+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000dee540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:25:38.45523+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:25:38.543538+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00098f180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:25:38.543582+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:25:42.464544+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000dee540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:42.465465+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f30700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:42.55588+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00098f180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:25:48.465443+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000dee540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:48.465892+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f30700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:48.556534+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00098f180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1714130748.693293,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002e776c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714130748.6933434,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714130748.7772105,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003392fc0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714130748.777252,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1714130749.6235354,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00256fa40/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714130749.6235964,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:25:54.466101+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000dee540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:54.467997+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f30700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:25:54.557634+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00098f180/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ jq -r .error_code ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68a016540002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:19925, start at 2024-04-26 19:26:21.58992993 +0800 CST m=+5.561416386 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:28:21.597 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:26:21.589 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:16:21.589 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68a016540002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:19925, start at 2024-04-26 19:26:21.58992993 +0800 CST m=+5.561416386 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:28:21.597 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:26:21.589 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:16:21.589 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68a017bc0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:20004, start at 2024-04-26 19:26:21.682437377 +0800 CST m=+5.607784528 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:28:21.689 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:26:21.679 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:16:21.679 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.21245.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349100945539073 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349100945539073 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349100945539073 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:26:28 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2127921281.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:26:31 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/54df52ba-ab7e-4163-bfaf-c3921e2b3735 {"id":"54df52ba-ab7e-4163-bfaf-c3921e2b3735","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a27d80bf5 54df52ba-ab7e-4163-bfaf-c3921e2b3735 /tidb/cdc/default/default/upstream/7362135606850409708 {"id":7362135606850409708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/54df52ba-ab7e-4163-bfaf-c3921e2b3735 {"id":"54df52ba-ab7e-4163-bfaf-c3921e2b3735","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a27d80bf5 54df52ba-ab7e-4163-bfaf-c3921e2b3735 /tidb/cdc/default/default/upstream/7362135606850409708 {"id":7362135606850409708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/54df52ba-ab7e-4163-bfaf-c3921e2b3735 {"id":"54df52ba-ab7e-4163-bfaf-c3921e2b3735","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a27d80bf5 54df52ba-ab7e-4163-bfaf-c3921e2b3735 /tidb/cdc/default/default/upstream/7362135606850409708 {"id":7362135606850409708,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349100945539073 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.21324.out cli changefeed create --start-ts=449349100945539073 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362135606850409708,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:26:31.700392051+08:00","start_ts":449349100945539073,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349100945539073,"checkpoint_ts":449349100945539073,"checkpoint_time":"2024-04-26 19:26:26.688"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 19306 24.7 0.4 3811288 1686836 ? Sl 19:26 0:06 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 19307 16.2 0.4 3761112 1616168 ? Sl 19:26 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 19308 16.2 0.4 3783124 1638816 ? Sl 19:26 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 19322 23.3 0.4 3804628 1676472 ? Sl 19:26 0:06 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3140 0 --:--:-- --:--:-- --:--:-- 3155 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:26:40.089","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:26:33.738","now_ts":"2024-04-26 19:26:40.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:26:40.089","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:26:33.738","now_ts":"2024-04-26' '19:26:40.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:26:40.089","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:26:33.738","now_ts":"2024-04-26' '19:26:40.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 6670 0 --:--:-- --:--:-- --:--:-- 6694 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:26:40.089","puller_resolved_ts":"2024-04-26 19:26:40.089","last_synced_ts":"2024-04-26 19:26:33.738","now_ts":"2024-04-26 19:28:51.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:26:40.089","puller_resolved_ts":"2024-04-26' '19:26:40.089","last_synced_ts":"2024-04-26' '19:26:33.738","now_ts":"2024-04-26' '19:28:51.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:26:40.089","puller_resolved_ts":"2024-04-26' '19:26:40.089","last_synced_ts":"2024-04-26' '19:26:33.738","now_ts":"2024-04-26' '19:28:51.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68aaa5c0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:22475, start at 2024-04-26 19:29:14.625845771 +0800 CST m=+5.247920240 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:31:14.634 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:29:14.608 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:19:14.608 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68aaa5c0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:22475, start at 2024-04-26 19:29:14.625845771 +0800 CST m=+5.247920240 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:31:14.634 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:29:14.608 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:19:14.608 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68aaa7240003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:22563, start at 2024-04-26 19:29:14.699278005 +0800 CST m=+5.252065526 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:31:14.707 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:29:14.697 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:19:14.697 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.23814.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349145868894209 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349145868894209 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349145868894209 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:29:19 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2384423846.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:29:22 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d68b9adc-deb8-4ce1-ac76-e60abd4a4b85 {"id":"d68b9adc-deb8-4ce1-ac76-e60abd4a4b85","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2a7d0ef6 d68b9adc-deb8-4ce1-ac76-e60abd4a4b85 /tidb/cdc/default/default/upstream/7362136345726971162 {"id":7362136345726971162,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d68b9adc-deb8-4ce1-ac76-e60abd4a4b85 {"id":"d68b9adc-deb8-4ce1-ac76-e60abd4a4b85","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2a7d0ef6 d68b9adc-deb8-4ce1-ac76-e60abd4a4b85 /tidb/cdc/default/default/upstream/7362136345726971162 {"id":7362136345726971162,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d68b9adc-deb8-4ce1-ac76-e60abd4a4b85 {"id":"d68b9adc-deb8-4ce1-ac76-e60abd4a4b85","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2a7d0ef6 d68b9adc-deb8-4ce1-ac76-e60abd4a4b85 /tidb/cdc/default/default/upstream/7362136345726971162 {"id":7362136345726971162,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349145868894209 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.23890.out cli changefeed create --start-ts=449349145868894209 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362136345726971162,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:29:22.935847171+08:00","start_ts":449349145868894209,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349145868894209,"checkpoint_ts":449349145868894209,"checkpoint_time":"2024-04-26 19:29:18.057"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 22475 11.6 0.0 2591116 183716 ? Sl 19:29 0:02 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714130949371544161.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 22479 3.6 0.0 2541396 163628 ? Sl 19:29 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714130949374384305.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 22563 12.8 0.0 3010040 209052 ? Sl 19:29 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714130949439282625.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3656 0 --:--:-- --:--:-- --:--:-- 3681 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:29:30.458","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:29:24.408","now_ts":"2024-04-26 19:29:31.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:29:30.458","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:29:24.408","now_ts":"2024-04-26' '19:29:31.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:29:30.458","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:29:24.408","now_ts":"2024-04-26' '19:29:31.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2656 0 --:--:-- --:--:-- --:--:-- 2662 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:31:40.607","puller_resolved_ts":"2024-04-26 19:31:33.558","last_synced_ts":"2024-04-26 19:29:24.408","now_ts":"2024-04-26 19:31:41.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:31:40.607","puller_resolved_ts":"2024-04-26' '19:31:33.558","last_synced_ts":"2024-04-26' '19:29:24.408","now_ts":"2024-04-26' '19:31:41.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:31:40.607","puller_resolved_ts":"2024-04-26' '19:31:33.558","last_synced_ts":"2024-04-26' '19:29:24.408","now_ts":"2024-04-26' '19:31:41.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b5007c001c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:24957, start at 2024-04-26 19:32:04.297942028 +0800 CST m=+5.200838674 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:34:04.304 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:32:04.305 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:22:04.305 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b5007c001c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:24957, start at 2024-04-26 19:32:04.297942028 +0800 CST m=+5.200838674 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:34:04.304 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:32:04.305 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:22:04.305 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b501340014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:25024, start at 2024-04-26 19:32:04.333158658 +0800 CST m=+5.176645847 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:34:04.341 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:32:04.301 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:22:04.301 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26307.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349190327992321 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349190327992321 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349190327992321 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:32:09 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2634326345.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:32:12 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2bbc8acd-00da-4def-a508-05a1a470fd2f {"id":"2bbc8acd-00da-4def-a508-05a1a470fd2f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2d161bf4 2bbc8acd-00da-4def-a508-05a1a470fd2f /tidb/cdc/default/default/upstream/7362137086165124283 {"id":7362137086165124283,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2bbc8acd-00da-4def-a508-05a1a470fd2f {"id":"2bbc8acd-00da-4def-a508-05a1a470fd2f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2d161bf4 2bbc8acd-00da-4def-a508-05a1a470fd2f /tidb/cdc/default/default/upstream/7362137086165124283 {"id":7362137086165124283,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2bbc8acd-00da-4def-a508-05a1a470fd2f {"id":"2bbc8acd-00da-4def-a508-05a1a470fd2f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2d161bf4 2bbc8acd-00da-4def-a508-05a1a470fd2f /tidb/cdc/default/default/upstream/7362137086165124283 {"id":7362137086165124283,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349190327992321 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.26398.out cli changefeed create --start-ts=449349190327992321 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362137086165124283,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:32:12.568570744+08:00","start_ts":449349190327992321,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349190327992321,"checkpoint_ts":449349190327992321,"checkpoint_time":"2024-04-26 19:32:07.655"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 7394 0 --:--:-- --:--:-- --:--:-- 7453 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:32:07.655","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-04-26 19:32:34.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:32:07.655","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:32:34.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:32:07.655","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:32:34.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b85ba00003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:27388, start at 2024-04-26 19:32:59.241133645 +0800 CST m=+5.271113271 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:34:59.247 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:32:59.240 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:22:59.240 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b85ba00003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:27388, start at 2024-04-26 19:32:59.241133645 +0800 CST m=+5.271113271 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:34:59.247 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:32:59.240 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:22:59.240 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b85a000014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:27446, start at 2024-04-26 19:32:59.174036828 +0800 CST m=+5.139662908 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:34:59.180 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:32:59.186 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:22:59.186 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.28692.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349205253160961 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349205253160961 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349205253160961 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:33:06 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2871928721.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:33:09 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f {"id":"73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2dea59f7 73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f /tidb/cdc/default/default/upstream/7362137307887726620 {"id":7362137307887726620,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f {"id":"73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2dea59f7 73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f /tidb/cdc/default/default/upstream/7362137307887726620 {"id":7362137307887726620,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f {"id":"73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2dea59f7 73ec6140-bc3b-4c5a-b6ef-8d12fac0cd3f /tidb/cdc/default/default/upstream/7362137307887726620 {"id":7362137307887726620,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349205253160961 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.28776.out cli changefeed create --start-ts=449349205253160961 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362137307887726620,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:33:09.475131455+08:00","start_ts":449349205253160961,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349205253160961,"checkpoint_ts":449349205253160961,"checkpoint_time":"2024-04-26 19:33:04.590"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2275 0 --:--:-- --:--:-- --:--:-- 2278 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:33:04.590","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-04-26 19:33:10.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:33:04.590","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:33:10.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:33:04.590","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:33:10.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-04-26 19:33:04.590' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:33:04.590","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:33:10.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:33:04.590","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:33:10.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-04-26 19:33:11' + echo 'sink_checkpoint_ts is 2024-04-26' 19:33:04.590 sink_checkpoint_ts is 2024-04-26 19:33:04.590 ++ date -d '2024-04-26 19:33:04.590' +%s + checkpoint_timestamp=1714131184 ++ date -d '2024-04-26 19:33:11' +%s + current_timestamp=1714131191 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2357 0 --:--:-- --:--:-- --:--:-- 2382 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:33:17.040","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:33:11.640","now_ts":"2024-04-26 19:33:18.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:33:17.040","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:33:11.640","now_ts":"2024-04-26' '19:33:18.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:33:17.040","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:33:11.640","now_ts":"2024-04-26' '19:33:18.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2662 0 --:--:-- --:--:-- --:--:-- 2695 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:35:27.190","puller_resolved_ts":"2024-04-26 19:35:21.190","last_synced_ts":"2024-04-26 19:33:11.640","now_ts":"2024-04-26 19:35:28.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:35:27.190","puller_resolved_ts":"2024-04-26' '19:35:21.190","last_synced_ts":"2024-04-26' '19:33:11.640","now_ts":"2024-04-26' '19:35:28.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 26671 7.3 0.0 13781004 138676 ? Sl 19:32 0:11 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 26717 4.9 0.0 13911756 133828 ? Sl 19:32 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1714131334.2046316,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002678000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714131334.2047048,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714131334.2721858,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002e7dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714131334.2722497,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714131335.2219548,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022956c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714131335.222003,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:35:39.051332+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:39.051801+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:39.098531+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b2f500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:45.052609+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:45.05312+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:45.099382+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b2f500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:35:51.053838+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:51.054121+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:51.100153+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b2f500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:35:57.054811+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:57.055632+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:35:57.10113+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b2f500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:35:59.042905+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:35:59.042948+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:35:59.043296+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:35:59.043337+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:35:59.088306+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b2f500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:35:59.088338+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:36:03.055866+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:36:03.056922+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:36:03.101738+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b2f500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:36:09.057752+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:36:09.058323+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:36:09.103172+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b2f500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1714131369.205969,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002678000/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714131369.205995,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714131369.272956,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002e7dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714131369.272979,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0{"level":"warn","ts":1714131370.222831,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0022956c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714131370.22288,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:36:15.059444+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f60e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:36:15.059448+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e8c540/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:36:15.104484+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b2f500/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ jq -r .error_code ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68c5dc240017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:29970, start at 2024-04-26 19:36:40.506681628 +0800 CST m=+5.103800223 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:38:40.514 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:36:40.507 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:26:40.507 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68c5dc240017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:29970, start at 2024-04-26 19:36:40.506681628 +0800 CST m=+5.103800223 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:38:40.514 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:36:40.507 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:26:40.507 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68c5dcd80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:30045, start at 2024-04-26 19:36:40.52400447 +0800 CST m=+5.071992433 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:38:40.530 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:36:40.502 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:26:40.502 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.31280.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349262759165953 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349262759165953 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349262759165953 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:36:45 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3132131323.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:36:48 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/926487e6-2fe2-4ce5-97c9-6ecf586aa663 {"id":"926487e6-2fe2-4ce5-97c9-6ecf586aa663","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a314f5ef6 926487e6-2fe2-4ce5-97c9-6ecf586aa663 /tidb/cdc/default/default/upstream/7362138273236505143 {"id":7362138273236505143,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/926487e6-2fe2-4ce5-97c9-6ecf586aa663 {"id":"926487e6-2fe2-4ce5-97c9-6ecf586aa663","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a314f5ef6 926487e6-2fe2-4ce5-97c9-6ecf586aa663 /tidb/cdc/default/default/upstream/7362138273236505143 {"id":7362138273236505143,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/926487e6-2fe2-4ce5-97c9-6ecf586aa663 {"id":"926487e6-2fe2-4ce5-97c9-6ecf586aa663","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a314f5ef6 926487e6-2fe2-4ce5-97c9-6ecf586aa663 /tidb/cdc/default/default/upstream/7362138273236505143 {"id":7362138273236505143,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349262759165953 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.31368.out cli changefeed create --start-ts=449349262759165953 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362138273236505143,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:36:48.867716945+08:00","start_ts":449349262759165953,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349262759165953,"checkpoint_ts":449349262759165953,"checkpoint_time":"2024-04-26 19:36:43.958"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 29349 23.6 0.4 3817428 1686496 ? Sl 19:36 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 29350 17.0 0.4 3778008 1640024 ? Sl 19:36 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 29351 16.7 0.4 3772372 1611492 ? Sl 19:36 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 29353 22.1 0.4 3797464 1675228 ? Sl 19:36 0:05 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2575 0 --:--:-- --:--:-- --:--:-- 2585 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:36:55.458","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:36:50.808","now_ts":"2024-04-26 19:36:57.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:36:55.458","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:36:50.808","now_ts":"2024-04-26' '19:36:57.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:36:55.458","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:36:50.808","now_ts":"2024-04-26' '19:36:57.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8807 0 --:--:-- --:--:-- --:--:-- 8925 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:36:56.458","puller_resolved_ts":"2024-04-26 19:36:57.458","last_synced_ts":"2024-04-26 19:36:50.808","now_ts":"2024-04-26 19:39:08.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:36:56.458","puller_resolved_ts":"2024-04-26' '19:36:57.458","last_synced_ts":"2024-04-26' '19:36:50.808","now_ts":"2024-04-26' '19:39:08.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ jq -r .info ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:36:56.458","puller_resolved_ts":"2024-04-26' '19:36:57.458","last_synced_ts":"2024-04-26' '19:36:50.808","now_ts":"2024-04-26' '19:39:08.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68d03aac0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:32397, start at 2024-04-26 19:39:30.36965045 +0800 CST m=+5.033924959 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:41:30.379 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:39:30.347 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:29:30.347 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68d03aac0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:32397, start at 2024-04-26 19:39:30.36965045 +0800 CST m=+5.033924959 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:41:30.379 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:39:30.347 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:29:30.347 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68d03ce40013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:32480, start at 2024-04-26 19:39:30.519911932 +0800 CST m=+5.131858846 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:41:30.526 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:39:30.489 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:29:30.489 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.33832.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349307307917316 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349307307917316 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349307307917316 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:39:35 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3386933871.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:39:38 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/225c1151-f3b2-4b0a-855f-6deebe50ff6f {"id":"225c1151-f3b2-4b0a-855f-6deebe50ff6f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a33e53bf3 225c1151-f3b2-4b0a-855f-6deebe50ff6f /tidb/cdc/default/default/upstream/7362139000328943588 {"id":7362139000328943588,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/225c1151-f3b2-4b0a-855f-6deebe50ff6f {"id":"225c1151-f3b2-4b0a-855f-6deebe50ff6f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a33e53bf3 225c1151-f3b2-4b0a-855f-6deebe50ff6f /tidb/cdc/default/default/upstream/7362139000328943588 {"id":7362139000328943588,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/225c1151-f3b2-4b0a-855f-6deebe50ff6f {"id":"225c1151-f3b2-4b0a-855f-6deebe50ff6f","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a33e53bf3 225c1151-f3b2-4b0a-855f-6deebe50ff6f /tidb/cdc/default/default/upstream/7362139000328943588 {"id":7362139000328943588,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349307307917316 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.33921.out cli changefeed create --start-ts=449349307307917316 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362139000328943588,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:39:38.84155919+08:00","start_ts":449349307307917316,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349307307917316,"checkpoint_ts":449349307307917316,"checkpoint_time":"2024-04-26 19:39:33.898"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 32397 11.0 0.0 2583016 213880 ? Sl 19:39 0:02 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714131565329236785.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 32401 3.4 0.0 2655884 162224 ? Sl 19:39 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714131565332189079.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 32480 12.1 0.0 3018236 209620 ? Sl 19:39 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714131565380669403.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 1646 0 --:--:-- --:--:-- --:--:-- 1641 100 243 100 243 0 0 1645 0 --:--:-- --:--:-- --:--:-- 1641 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:39:46.398","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:39:40.347","now_ts":"2024-04-26 19:39:47.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:39:46.398","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:39:40.347","now_ts":"2024-04-26' '19:39:47.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:39:46.398","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:39:40.347","now_ts":"2024-04-26' '19:39:47.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2572 0 --:--:-- --:--:-- --:--:-- 2600 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:41:56.497","puller_resolved_ts":"2024-04-26 19:41:50.497","last_synced_ts":"2024-04-26 19:39:40.347","now_ts":"2024-04-26 19:41:57.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:41:56.497","puller_resolved_ts":"2024-04-26' '19:41:50.497","last_synced_ts":"2024-04-26' '19:39:40.347","now_ts":"2024-04-26' '19:41:57.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:41:56.497","puller_resolved_ts":"2024-04-26' '19:41:50.497","last_synced_ts":"2024-04-26' '19:39:40.347","now_ts":"2024-04-26' '19:41:57.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68da85600013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:35002, start at 2024-04-26 19:42:19.009110939 +0800 CST m=+5.154075893 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:44:19.015 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:42:19.018 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:32:19.018 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68da85600013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:35002, start at 2024-04-26 19:42:19.009110939 +0800 CST m=+5.154075893 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:44:19.015 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:42:19.018 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:32:19.018 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68da878c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:xtiflow-release-7-5-pull-cdc-integration-storage-test-285-kq0s5, pid:35084, start at 2024-04-26 19:42:19.123254258 +0800 CST m=+5.190707953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:44:19.131 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:42:19.107 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:32:19.107 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.36378.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349351497531393 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349351497531393 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349351497531393 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:42:23 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3641536417.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:42:26 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/82378a1f-3326-4a4e-9099-8401a1b33437 {"id":"82378a1f-3326-4a4e-9099-8401a1b33437","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a367b6cf4 82378a1f-3326-4a4e-9099-8401a1b33437 /tidb/cdc/default/default/upstream/7362139724394471348 {"id":7362139724394471348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/82378a1f-3326-4a4e-9099-8401a1b33437 {"id":"82378a1f-3326-4a4e-9099-8401a1b33437","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a367b6cf4 82378a1f-3326-4a4e-9099-8401a1b33437 /tidb/cdc/default/default/upstream/7362139724394471348 {"id":7362139724394471348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/82378a1f-3326-4a4e-9099-8401a1b33437 {"id":"82378a1f-3326-4a4e-9099-8401a1b33437","address":"127.0.0.1:8300","version":"v7.5.1-57-g4a2af7498"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a367b6cf4 82378a1f-3326-4a4e-9099-8401a1b33437 /tidb/cdc/default/default/upstream/7362139724394471348 {"id":7362139724394471348,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349351497531393 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.36472.out cli changefeed create --start-ts=449349351497531393 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_storage_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362139724394471348,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:42:27.38284173+08:00","start_ts":449349351497531393,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g4a2af7498","resolved_ts":449349351497531393,"checkpoint_ts":449349351497531393,"checkpoint_time":"2024-04-26 19:42:22.468"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 7540 0 --:--:-- --:--:-- --:--:-- 7610 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:42:22.468","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-04-26 19:42:48.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:42:22.468","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:42:48.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:42:22.468","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:42:48.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Fri Apr 26 19:42:59 CST 2024] <<<<<< run test case synced_status success! >>>>>>' [Fri Apr 26 19:42:59 CST 2024] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_storage_test-285/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS