Started by user Jenkins Admin Obtained pipelines/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test.groovy from git https://github.com/PingCAP-QE/ci.git Loading library tipipeline@main Library tipipeline@main is cached. Copying from home. [Pipeline] Start of Pipeline [Pipeline] readJSON [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-79g2l-s2kl1 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "02bcc320039745d1ce8227372c7a028ff35d1d2d" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-79g2l" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout The recommended git tool is: git No credentials specified Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git rev-list --no-walk cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] timeout Timeout set to expire in 1 hr 0 min [Pipeline] { [Pipeline] stage [Pipeline] { (Debug info) [Pipeline] sh + printenv PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=ba921bbf-bb71-489d-a9a2-27bc75917704 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Debug info BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct _=/usr/bin/printenv POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test HUDSON_URL=https://do.pingcap.net/jenkins/ JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=3 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-79g2l GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-d4c91f37a2cd54d8a1ba68aedcc7e1b173b616003bd6c892b8b1a5e40c8eb9f7 NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0 ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-79g2l GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz + echo ------------------------- ------------------------- + go env GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/home/jenkins/.cache/go-build' GOENV='/home/jenkins/.config/go/env' GOEXE='' GOEXPERIMENT='' GOFLAGS='' GOHOSTARCH='amd64' GOHOSTOS='linux' GOINSECURE='' GOMODCACHE='/go/pkg/mod' GONOPROXY='' GONOSUMDB='' GOOS='linux' GOPATH='/go' GOPRIVATE='' GOPROXY='http://goproxy.apps.svc,https://proxy.golang.org,direct' GOROOT='/usr/local/go' GOSUMDB='sum.golang.org' GOTMPDIR='' GOTOOLCHAIN='auto' GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64' GOVCS='' GOVERSION='go1.21.6' GCCGO='gccgo' GOAMD64='v1' AR='ar' CC='gcc' CXX='g++' CGO_ENABLED='1' GOMOD='/dev/null' GOWORK='' CGO_CFLAGS='-O2 -g' CGO_CPPFLAGS='' CGO_CXXFLAGS='-O2 -g' CGO_FFLAGS='-O2 -g' CGO_LDFLAGS='-O2 -g' PKG_CONFIG='pkg-config' GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1895018526=/tmp/go-build -gno-record-gcc-switches' + echo ------------------------- ------------------------- + echo 'debug command: kubectl -n jenkins-tiflow exec -ti ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0 bash' debug command: kubectl -n jenkins-tiflow exec -ti ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-q8zb0 bash [Pipeline] container [Pipeline] { [Pipeline] sh + dig github.com ; <<>> DiG 9.18.16 <<>> github.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39983 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: 71f055cef44cdec0 (echoed) ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: github.com. 18 IN A 20.205.243.166 ;; Query time: 0 msec ;; SERVER: 169.254.25.10#53(169.254.25.10) (UDP) ;; WHEN: Fri Apr 26 10:56:21 UTC 2024 ;; MSG SIZE rcvd: 77 [Pipeline] script [Pipeline] { [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Check diff files) [Pipeline] container [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $token [Pipeline] { [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=1&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=1&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=2&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=2&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=3&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=3&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=4&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=4&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] httpRequest Warning: A secret was passed to "httpRequest" using Groovy String interpolation, which is insecure. Affected argument(s) used the following variable(s): [token] See https://jenkins.io/redirect/groovy-string-interpolation for details. HttpMethod: GET URL: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=5&per_page=100 Content-Type: application/json Authorization: ***** Sending request to url: https://api.github.com/repos/pingcap/tiflow/pulls/10970/files?page=5&per_page=100 Response Code: HTTP/1.1 200 OK Success: Status code 200 is in the accepted range: 100:399 [Pipeline] } [Pipeline] // withCredentials [Pipeline] echo pr_diff_files: [Makefile, cdc/api/v2/api_helpers.go, cdc/api/v2/api_helpers_mock.go, cdc/api/v2/changefeed.go, cdc/api/v2/changefeed_test.go, cdc/api/v2/model.go, cdc/api/v2/model_test.go, cdc/api/v2/unsafe.go, cdc/entry/codec.go, cdc/entry/codec_test.go, cdc/entry/mounter.go, cdc/entry/mounter_test.go, cdc/entry/schema/snapshot.go, cdc/entry/schema/snapshot_test.go, cdc/entry/schema_storage.go, cdc/entry/schema_storage_test.go, cdc/entry/schema_test_helper.go, cdc/entry/validator.go, cdc/kv/client_bench_test.go, cdc/kv/client_test.go, cdc/kv/shared_client_test.go, cdc/kv/store_op.go, cdc/model/changefeed_test.go, cdc/model/codec/codec.go, cdc/model/codec/codec_test.go, cdc/model/codec/v1/codec.go, cdc/model/codec/v1/convert.go, cdc/model/codec/v1/convert_test.go, cdc/model/mounter.go, cdc/model/owner.go, cdc/model/schema_storage.go, cdc/model/schema_storage_test.go, cdc/model/sink.go, cdc/model/sink_test.go, cdc/owner/changefeed.go, cdc/owner/changefeed_test.go, cdc/owner/ddl_manager.go, cdc/owner/ddl_manager_test.go, cdc/owner/ddl_sink.go, cdc/owner/schema.go, cdc/owner/schema_test.go, cdc/processor/sourcemanager/engine/factory/factory.go, cdc/processor/sourcemanager/engine/factory/pebble.go, cdc/processor/sourcemanager/engine/pebble/db.go, cdc/processor/sourcemanager/engine/pebble/db_test.go, cdc/processor/sourcemanager/engine/pebble/event_sorter.go, cdc/processor/sourcemanager/engine/pebble/event_sorter_test.go, cdc/puller/ddl_puller.go, cdc/puller/ddl_puller_test.go, cdc/puller/puller.go, cdc/puller/puller_test.go, cdc/server/server.go, cdc/sink/ddlsink/cloudstorage/cloud_storage_ddl_sink.go, cdc/sink/ddlsink/cloudstorage/cloud_storage_ddl_sink_test.go, cdc/sink/ddlsink/mq/mq_ddl_sink_test.go, cdc/sink/ddlsink/mq/pulsar_ddl_sink_test.go, cdc/sink/ddlsink/mysql/mysql_ddl_sink.go, cdc/sink/ddlsink/mysql/mysql_ddl_sink_test.go, cdc/sink/dmlsink/cloudstorage/cloud_storage_dml_sink_test.go, cdc/sink/dmlsink/cloudstorage/defragmenter_test.go, cdc/sink/dmlsink/cloudstorage/dml_worker_test.go, cdc/sink/dmlsink/cloudstorage/encoding_worker_test.go, cdc/sink/dmlsink/mq/dispatcher/event_router.go, cdc/sink/dmlsink/mq/dispatcher/event_router_test.go, cdc/sink/dmlsink/mq/dispatcher/partition/columns_test.go, cdc/sink/dmlsink/mq/dispatcher/partition/index_value_test.go, cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go, cdc/sink/dmlsink/txn/event.go, cdc/sink/dmlsink/txn/event_test.go, cdc/sink/dmlsink/txn/mysql/dml.go, cdc/sink/dmlsink/txn/mysql/dml_test.go, cdc/sink/dmlsink/txn/mysql/mysql.go, cdc/sink/dmlsink/txn/mysql/mysql_test.go, cmd/cdc/main.go, cmd/dm-syncer/config.go, cmd/dm-worker/main.go, cmd/filter-helper/main.go, cmd/kafka-consumer/main.go, dm/chaos/cases/db.go, dm/chaos/cases/diff.go, dm/chaos/cases/schema.go, dm/chaos/cases/stmt.go, dm/chaos/cases/task.go, dm/checker/check_test.go, dm/checker/checker.go, dm/config/subtask.go, dm/config/subtask_test.go, dm/config/task.go, dm/config/task_converters.go, dm/config/task_converters_test.go, dm/config/task_test.go, dm/ctl/common/util.go, dm/dumpling/dumpling.go, dm/dumpling/dumpling_test.go, dm/loader/checkpoint.go, dm/loader/lightning.go, dm/loader/lightning_test.go, dm/master/config.go, dm/master/config_test.go, dm/master/openapi_view.go, dm/master/server.go, dm/master/server_test.go, dm/master/shardddl/optimist.go, dm/master/shardddl/optimist_test.go, dm/pkg/binlog/event/util.go, dm/pkg/checker/binlog.go, dm/pkg/checker/binlog_test.go, dm/pkg/checker/conn_checker.go, dm/pkg/checker/lightning.go, dm/pkg/checker/lightning_test.go, dm/pkg/checker/mysql_server.go, dm/pkg/checker/mysql_server_test.go, dm/pkg/checker/onlineddl.go, dm/pkg/checker/privilege.go, dm/pkg/checker/privilege_test.go, dm/pkg/checker/table_structure.go, dm/pkg/checker/table_structure_test.go, dm/pkg/checker/utils.go, dm/pkg/checker/utils_test.go, dm/pkg/conn/baseconn.go, dm/pkg/conn/baseconn_test.go, dm/pkg/conn/basedb.go, dm/pkg/conn/db.go, dm/pkg/conn/db_test.go, dm/pkg/conn/mockdb.go, dm/pkg/conn/utils.go, dm/pkg/election/election_test.go, dm/pkg/etcdutil/etcdutil_test.go, dm/pkg/log/log.go, dm/pkg/log/log_test.go, dm/pkg/parser/common.go, dm/pkg/parser/common_test.go, dm/pkg/retry/errors.go, dm/pkg/retry/strategy_test.go, dm/pkg/schema/tracker.go, dm/pkg/schema/tracker_test.go, dm/pkg/schema/visitor.go, dm/pkg/shardddl/optimism/info.go, dm/pkg/shardddl/optimism/info_test.go, dm/pkg/shardddl/optimism/keeper.go, dm/pkg/shardddl/optimism/keeper_test.go, dm/pkg/shardddl/optimism/lock.go, dm/pkg/shardddl/optimism/lock_test.go, dm/pkg/shardddl/pessimism/info_test.go, dm/pkg/upgrade/upgrade.go, dm/pkg/utils/common.go, dm/pkg/utils/common_test.go, dm/pkg/utils/time.go, dm/pkg/utils/util.go, dm/pkg/utils/util_test.go, dm/pkg/v1dbschema/schema.go, dm/relay/file_util.go, dm/relay/file_util_test.go, dm/relay/relay.go, dm/relay/relay_test.go, dm/simulator/config/config.go, dm/simulator/sqlgen/impl.go, dm/simulator/sqlgen/impl_test.go, dm/syncer/causality.go, dm/syncer/checkpoint.go, dm/syncer/checkpoint_flush_worker.go, dm/syncer/checkpoint_test.go, dm/syncer/compactor_test.go, dm/syncer/data_validator.go, dm/syncer/data_validator_test.go, dm/syncer/dbconn/db.go, dm/syncer/dbconn/upstream_db.go, dm/syncer/dbconn/utils.go, dm/syncer/ddl.go, dm/syncer/ddl_test.go, dm/syncer/dml.go, dm/syncer/dml_test.go, dm/syncer/dml_worker_test.go, dm/syncer/error.go, dm/syncer/error_test.go, dm/syncer/expr_filter_group.go, dm/syncer/expr_filter_group_test.go, dm/syncer/filter.go, dm/syncer/filter_test.go, dm/syncer/handle_error.go, dm/syncer/job.go, dm/syncer/job_test.go, dm/syncer/online-ddl-tools/online_ddl.go, dm/syncer/opt_sharding_group.go, dm/syncer/optimist.go, dm/syncer/safe-mode/mode.go, dm/syncer/safe-mode/mode_test.go, dm/syncer/schema.go, dm/syncer/shardddl/optimist.go, dm/syncer/shardddl/optimist_test.go, dm/syncer/sharding-meta/shardmeta.go, dm/syncer/sharding_group.go, dm/syncer/sharding_group_test.go, dm/syncer/status_test.go, dm/syncer/syncer.go, dm/syncer/syncer_test.go, dm/syncer/util.go, dm/syncer/util_test.go, dm/syncer/validate_worker.go, dm/syncer/validate_worker_test.go, dm/syncer/validator_checkpoint.go, dm/syncer/validator_checkpoint_test.go, dm/syncer/validator_cond.go, dm/syncer/validator_cond_test.go, dm/tests/incompatible_ddl_changes/conf/dm-task1.yaml, dm/tests/incompatible_ddl_changes/run.sh, dm/tests/lightning_mode/run.sh, dm/tests/tls/run.sh, dm/unit/unit_test.go, dm/worker/server_test.go, dm/worker/task_checker_test.go, engine/chaos/cases/dm/case.go, engine/chaos/cases/dm/db.go, engine/executor/server.go, engine/jobmaster/dm/checkpoint/agent.go, engine/jobmaster/dm/checkpoint/agent_test.go, engine/jobmaster/dm/config/config.go, engine/jobmaster/dm/ddl_coordinator.go, engine/pkg/meta/internal/etcdkv/mock.go, engine/pkg/meta/model/config.go, engine/pkg/orm/util_test.go, examples/golang/avro-checksum-verification/go.mod, examples/golang/avro-checksum-verification/go.sum, examples/golang/avro-checksum-verification/main.go, go.mod, go.sum, pkg/applier/redo.go, pkg/applier/redo_test.go, pkg/cmd/server/server.go, pkg/cmd/server/server_test.go, pkg/config/filter.go, pkg/config/outdated/v1.go, pkg/config/replica_config.go, pkg/errorutil/util.go, pkg/errorutil/util_test.go, pkg/etcd/etcd.go, pkg/filter/expr_filter.go, pkg/filter/expr_filter_test.go, pkg/filter/filter.go, pkg/filter/filter_test.go, pkg/filter/filter_test_helper.go, pkg/filter/sql_event_filter.go, pkg/filter/utils.go, pkg/filter/utils_test.go, pkg/httputil/httputil_test.go, pkg/migrate/migrate_test.go, pkg/pdutil/api_client.go, pkg/pdutil/api_client_test.go, pkg/pdutil/utils_test.go, pkg/sink/cloudstorage/path_test.go, pkg/sink/cloudstorage/table_definition.go, pkg/sink/cloudstorage/table_definition_test.go, pkg/sink/codec/avro/avro.go, pkg/sink/codec/avro/avro_test.go, pkg/sink/codec/avro/decoder.go, pkg/sink/codec/avro/decoder_test.go, pkg/sink/codec/canal/canal_encoder_test.go, pkg/sink/codec/canal/canal_entry.go, pkg/sink/codec/canal/canal_entry_test.go, pkg/sink/codec/canal/canal_json_message.go, pkg/sink/codec/canal/canal_json_row_event_encoder_test.go, pkg/sink/codec/canal/canal_json_txn_event_encoder_test.go, pkg/sink/codec/canal/canal_test_util.go, pkg/sink/codec/common/message_test.go, pkg/sink/codec/common/verify_checksum.go, pkg/sink/codec/craft/craft_encoder_test.go, pkg/sink/codec/craft/message_decoder.go, pkg/sink/codec/craft/message_encoder.go, pkg/sink/codec/csv/csv_decoder.go, pkg/sink/codec/csv/csv_decoder_test.go, pkg/sink/codec/csv/csv_encoder_test.go, pkg/sink/codec/csv/csv_message.go, pkg/sink/codec/csv/csv_message_test.go, pkg/sink/codec/internal/batch_tester.go, pkg/sink/codec/internal/column.go, pkg/sink/codec/internal/java.go, pkg/sink/codec/maxwell/maxwell_encoder_test.go, pkg/sink/codec/maxwell/maxwell_message.go, pkg/sink/codec/maxwell/maxwell_message_test.go, pkg/sink/codec/open/open_protocol_decoder.go, pkg/sink/codec/open/open_protocol_encoder_test.go, pkg/sink/codec/open/open_protocol_message.go, pkg/sink/codec/open/open_protocol_message_test.go, pkg/sink/kafka/claimcheck/claim_check_test.go, pkg/sink/mysql/db_helper.go, pkg/spanz/span.go, pkg/spanz/span_test.go, pkg/sqlmodel/causality.go, pkg/sqlmodel/row_change.go, pkg/sqlmodel/row_change_test.go, pkg/sqlmodel/utils.go, pkg/sqlmodel/where_handle.go, pkg/sqlmodel/where_handle_test.go, pkg/txnutil/gc/gc_manager.go, pkg/txnutil/gc/gc_manager_test.go, pkg/upstream/upstream.go, pkg/util/memory.go, pkg/util/tz.go, pkg/version/check.go, pkg/version/check_test.go, tests/integration_tests/cdc/dailytest/db.go, tests/integration_tests/cdc/dailytest/job.go, tests/integration_tests/cdc/dailytest/parser.go, tests/integration_tests/resolve_lock/main.go, tests/integration_tests/util/db.go] [Pipeline] echo diff file not matched: Makefile [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Checkout) [Pipeline] timeout Timeout set to expire in 10 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache Cache restored successfully (git/pingcap/tiflow/rev-823a389-c68c96e) 200006144 bytes in 1.43 secs (140293499 bytes/sec) [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] script [Pipeline] { [Pipeline] sh git version 2.36.6 Reinitialized existing Git repository in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/.git/ .git HEAD is now at c68c96e01 sink(ticdc): limit the number of transactions cached in a mysql worker (#10892) POST git-upload-pack (686 bytes) POST git-upload-pack (973 bytes) From https://github.com/pingcap/tiflow = [up to date] release-7.5 -> origin/release-7.5 13c5e9410..aa323aa99 refs/pull/10970/head -> origin/pr/10970/head Previous HEAD position was c68c96e01 sink(ticdc): limit the number of transactions cached in a mysql worker (#10892) HEAD is now at 823a3899c sink(ticdc): Revert changes related to the conflict detector (#10924) ๐Ÿšง Checkouting to base SHA:823a3899cc17bf14ad7875ab0cc69092186abc4a... HEAD is now at 823a3899c sink(ticdc): Revert changes related to the conflict detector (#10924) โœ… Checked. ๐ŸŽ‰ ๐Ÿงพ HEAD info: 823a3899cc17bf14ad7875ab0cc69092186abc4a 823a3899c sink(ticdc): Revert changes related to the conflict detector (#10924) 3c4a12d87 mysql(ticdc): fix the data race in the conflict detector (#10630) (#10642) 0d6a6613d kvclient(ticdc): close the grpc client after all goroutine exited to prevent data race and panic (#10865) (#10906) ๐Ÿšง Pre-merge heads of pull requests to base SHA: 823a3899cc17bf14ad7875ab0cc69092186abc4a ... Merge made by the 'ort' strategy. Makefile | 2 +- cdc/api/v2/api_helpers.go | 2 +- cdc/api/v2/api_helpers_mock.go | 2 +- cdc/api/v2/changefeed.go | 2 +- cdc/api/v2/changefeed_test.go | 2 +- cdc/api/v2/model.go | 2 +- cdc/api/v2/model_test.go | 2 +- cdc/api/v2/unsafe.go | 2 +- cdc/entry/codec.go | 12 +- cdc/entry/codec_test.go | 6 +- cdc/entry/mounter.go | 38 +- cdc/entry/mounter_test.go | 60 +- cdc/entry/schema/snapshot.go | 4 +- cdc/entry/schema/snapshot_test.go | 2 +- cdc/entry/schema_storage.go | 4 +- cdc/entry/schema_storage_test.go | 24 +- cdc/entry/schema_test_helper.go | 18 +- cdc/entry/validator.go | 2 +- cdc/kv/client_bench_test.go | 2 +- cdc/kv/client_test.go | 4 +- cdc/kv/shared_client_test.go | 2 +- cdc/kv/store_op.go | 6 +- cdc/model/changefeed_test.go | 2 +- cdc/model/codec/codec.go | 2 +- cdc/model/codec/codec_test.go | 6 +- cdc/model/codec/v1/codec.go | 4 +- cdc/model/codec/v1/convert.go | 2 +- cdc/model/codec/v1/convert_test.go | 6 +- cdc/model/mounter.go | 2 +- cdc/model/owner.go | 2 +- cdc/model/schema_storage.go | 10 +- cdc/model/schema_storage_test.go | 6 +- cdc/model/sink.go | 6 +- cdc/model/sink_test.go | 6 +- cdc/owner/changefeed.go | 2 +- cdc/owner/changefeed_test.go | 2 +- cdc/owner/ddl_manager.go | 2 +- cdc/owner/ddl_manager_test.go | 2 +- cdc/owner/ddl_sink.go | 6 +- cdc/owner/schema.go | 6 +- cdc/owner/schema_test.go | 21 +- .../sourcemanager/engine/factory/factory.go | 6 +- .../sourcemanager/engine/factory/pebble.go | 39 +- cdc/processor/sourcemanager/engine/pebble/db.go | 13 +- .../sourcemanager/engine/pebble/db_test.go | 1 + .../sourcemanager/engine/pebble/event_sorter.go | 6 +- .../engine/pebble/event_sorter_test.go | 8 +- cdc/puller/ddl_puller.go | 6 +- cdc/puller/ddl_puller_test.go | 6 +- cdc/puller/puller.go | 2 +- cdc/puller/puller_test.go | 4 +- cdc/server/server.go | 2 +- .../ddlsink/cloudstorage/cloud_storage_ddl_sink.go | 2 +- .../cloudstorage/cloud_storage_ddl_sink_test.go | 6 +- cdc/sink/ddlsink/mq/mq_ddl_sink_test.go | 2 +- cdc/sink/ddlsink/mq/pulsar_ddl_sink_test.go | 2 +- cdc/sink/ddlsink/mysql/mysql_ddl_sink.go | 2 +- cdc/sink/ddlsink/mysql/mysql_ddl_sink_test.go | 4 +- .../cloudstorage/cloud_storage_dml_sink_test.go | 8 +- cdc/sink/dmlsink/cloudstorage/defragmenter_test.go | 6 +- cdc/sink/dmlsink/cloudstorage/dml_worker_test.go | 6 +- .../dmlsink/cloudstorage/encoding_worker_test.go | 6 +- cdc/sink/dmlsink/mq/dispatcher/event_router.go | 2 +- .../dmlsink/mq/dispatcher/event_router_test.go | 2 +- .../mq/dispatcher/partition/columns_test.go | 2 +- .../mq/dispatcher/partition/index_value_test.go | 2 +- .../transformer/columnselector/column_selector.go | 2 +- cdc/sink/dmlsink/txn/event.go | 2 +- cdc/sink/dmlsink/txn/event_test.go | 2 +- cdc/sink/dmlsink/txn/mysql/dml.go | 2 +- cdc/sink/dmlsink/txn/mysql/dml_test.go | 4 +- cdc/sink/dmlsink/txn/mysql/mysql.go | 8 +- cdc/sink/dmlsink/txn/mysql/mysql_test.go | 12 +- cmd/cdc/main.go | 2 +- cmd/dm-syncer/config.go | 4 +- cmd/dm-worker/main.go | 3 + cmd/filter-helper/main.go | 2 +- cmd/kafka-consumer/main.go | 2 +- dm/chaos/cases/db.go | 4 +- dm/chaos/cases/diff.go | 2 +- dm/chaos/cases/schema.go | 4 +- dm/chaos/cases/stmt.go | 4 +- dm/chaos/cases/task.go | 4 +- dm/checker/check_test.go | 4 +- dm/checker/checker.go | 50 +- dm/config/subtask.go | 8 +- dm/config/subtask_test.go | 2 +- dm/config/task.go | 8 +- dm/config/task_converters.go | 4 +- dm/config/task_converters_test.go | 2 +- dm/config/task_test.go | 4 +- dm/ctl/common/util.go | 2 +- dm/dumpling/dumpling.go | 6 +- dm/dumpling/dumpling_test.go | 6 +- dm/loader/checkpoint.go | 2 +- dm/loader/lightning.go | 46 +- dm/loader/lightning_test.go | 4 +- dm/master/config.go | 10 +- dm/master/config_test.go | 16 +- dm/master/openapi_view.go | 2 +- dm/master/server.go | 2 +- dm/master/server_test.go | 14 +- dm/master/shardddl/optimist.go | 2 +- dm/master/shardddl/optimist_test.go | 14 +- dm/pkg/binlog/event/util.go | 6 +- dm/pkg/checker/binlog.go | 5 +- dm/pkg/checker/binlog_test.go | 2 +- dm/pkg/checker/conn_checker.go | 4 +- dm/pkg/checker/lightning.go | 20 +- dm/pkg/checker/lightning_test.go | 2 +- dm/pkg/checker/mysql_server.go | 2 +- dm/pkg/checker/mysql_server_test.go | 2 +- dm/pkg/checker/onlineddl.go | 4 +- dm/pkg/checker/privilege.go | 14 +- dm/pkg/checker/privilege_test.go | 4 +- dm/pkg/checker/table_structure.go | 15 +- dm/pkg/checker/table_structure_test.go | 2 +- dm/pkg/checker/utils.go | 6 +- dm/pkg/checker/utils_test.go | 4 +- dm/pkg/conn/baseconn.go | 2 +- dm/pkg/conn/baseconn_test.go | 2 +- dm/pkg/conn/basedb.go | 4 +- dm/pkg/conn/db.go | 10 +- dm/pkg/conn/db_test.go | 8 +- dm/pkg/conn/mockdb.go | 16 +- dm/pkg/conn/utils.go | 2 +- dm/pkg/election/election_test.go | 8 +- dm/pkg/etcdutil/etcdutil_test.go | 20 +- dm/pkg/log/log.go | 4 +- dm/pkg/log/log_test.go | 4 +- dm/pkg/parser/common.go | 12 +- dm/pkg/parser/common_test.go | 4 +- dm/pkg/retry/errors.go | 2 +- dm/pkg/retry/strategy_test.go | 2 +- dm/pkg/schema/tracker.go | 43 +- dm/pkg/schema/tracker_test.go | 14 +- dm/pkg/schema/visitor.go | 4 +- dm/pkg/shardddl/optimism/info.go | 4 +- dm/pkg/shardddl/optimism/info_test.go | 12 +- dm/pkg/shardddl/optimism/keeper.go | 2 +- dm/pkg/shardddl/optimism/keeper_test.go | 6 +- dm/pkg/shardddl/optimism/lock.go | 10 +- dm/pkg/shardddl/optimism/lock_test.go | 10 +- dm/pkg/shardddl/pessimism/info_test.go | 2 +- dm/pkg/upgrade/upgrade.go | 2 +- dm/pkg/utils/common.go | 41 +- dm/pkg/utils/common_test.go | 2 +- dm/pkg/utils/time.go | 6 +- dm/pkg/utils/util.go | 2 +- dm/pkg/utils/util_test.go | 2 +- dm/pkg/v1dbschema/schema.go | 4 +- dm/relay/file_util.go | 2 +- dm/relay/file_util_test.go | 2 +- dm/relay/relay.go | 5 +- dm/relay/relay_test.go | 2 +- dm/simulator/config/config.go | 2 +- dm/simulator/sqlgen/impl.go | 10 +- dm/simulator/sqlgen/impl_test.go | 4 +- dm/syncer/causality.go | 2 +- dm/syncer/checkpoint.go | 8 +- dm/syncer/checkpoint_flush_worker.go | 2 +- dm/syncer/checkpoint_test.go | 8 +- dm/syncer/compactor_test.go | 4 +- dm/syncer/data_validator.go | 4 +- dm/syncer/data_validator_test.go | 6 +- dm/syncer/dbconn/db.go | 2 +- dm/syncer/dbconn/upstream_db.go | 6 +- dm/syncer/dbconn/utils.go | 4 +- dm/syncer/ddl.go | 58 +- dm/syncer/ddl_test.go | 10 +- dm/syncer/dml.go | 12 +- dm/syncer/dml_test.go | 16 +- dm/syncer/dml_worker_test.go | 10 +- dm/syncer/error.go | 16 +- dm/syncer/error_test.go | 2 +- dm/syncer/expr_filter_group.go | 20 +- dm/syncer/expr_filter_group_test.go | 6 +- dm/syncer/filter.go | 2 +- dm/syncer/filter_test.go | 4 +- dm/syncer/handle_error.go | 4 +- dm/syncer/job.go | 2 +- dm/syncer/job_test.go | 2 +- dm/syncer/online-ddl-tools/online_ddl.go | 10 +- dm/syncer/opt_sharding_group.go | 2 +- dm/syncer/optimist.go | 2 +- dm/syncer/safe-mode/mode.go | 2 +- dm/syncer/safe-mode/mode_test.go | 2 +- dm/syncer/schema.go | 14 +- dm/syncer/shardddl/optimist.go | 4 +- dm/syncer/shardddl/optimist_test.go | 12 +- dm/syncer/sharding-meta/shardmeta.go | 4 +- dm/syncer/sharding_group.go | 4 +- dm/syncer/sharding_group_test.go | 4 +- dm/syncer/status_test.go | 4 +- dm/syncer/syncer.go | 18 +- dm/syncer/syncer_test.go | 14 +- dm/syncer/util.go | 7 +- dm/syncer/util_test.go | 10 +- dm/syncer/validate_worker.go | 8 +- dm/syncer/validate_worker_test.go | 10 +- dm/syncer/validator_checkpoint.go | 4 +- dm/syncer/validator_checkpoint_test.go | 6 +- dm/syncer/validator_cond.go | 2 +- dm/syncer/validator_cond_test.go | 10 +- .../incompatible_ddl_changes/conf/dm-task1.yaml | 28 +- dm/tests/incompatible_ddl_changes/run.sh | 96 +- dm/tests/lightning_mode/run.sh | 2 +- dm/tests/tls/run.sh | 9 +- dm/unit/unit_test.go | 4 +- dm/worker/server_test.go | 8 +- dm/worker/task_checker_test.go | 2 +- engine/chaos/cases/dm/case.go | 4 +- engine/chaos/cases/dm/db.go | 2 +- engine/executor/server.go | 4 +- engine/jobmaster/dm/checkpoint/agent.go | 14 +- engine/jobmaster/dm/checkpoint/agent_test.go | 7 +- engine/jobmaster/dm/config/config.go | 4 +- engine/jobmaster/dm/ddl_coordinator.go | 10 +- engine/pkg/meta/internal/etcdkv/mock.go | 4 +- engine/pkg/meta/model/config.go | 6 +- engine/pkg/orm/util_test.go | 2 +- examples/golang/avro-checksum-verification/go.mod | 79 +- examples/golang/avro-checksum-verification/go.sum | 311 +++-- examples/golang/avro-checksum-verification/main.go | 23 +- go.mod | 276 ++--- go.sum | 1227 ++++++-------------- pkg/applier/redo.go | 2 +- pkg/applier/redo_test.go | 2 +- pkg/cmd/server/server.go | 2 +- pkg/cmd/server/server_test.go | 2 +- pkg/config/filter.go | 2 +- pkg/config/outdated/v1.go | 2 +- pkg/config/replica_config.go | 2 +- pkg/errorutil/util.go | 8 +- pkg/errorutil/util_test.go | 4 +- pkg/etcd/etcd.go | 4 +- pkg/filter/expr_filter.go | 24 +- pkg/filter/expr_filter_test.go | 14 +- pkg/filter/filter.go | 4 +- pkg/filter/filter_test.go | 2 +- pkg/filter/filter_test_helper.go | 18 +- pkg/filter/sql_event_filter.go | 6 +- pkg/filter/utils.go | 8 +- pkg/filter/utils_test.go | 18 +- pkg/httputil/httputil_test.go | 2 +- pkg/migrate/migrate_test.go | 4 +- pkg/pdutil/api_client.go | 6 +- pkg/pdutil/api_client_test.go | 6 +- pkg/pdutil/utils_test.go | 6 +- pkg/sink/cloudstorage/path_test.go | 6 +- pkg/sink/cloudstorage/table_definition.go | 8 +- pkg/sink/cloudstorage/table_definition_test.go | 8 +- pkg/sink/codec/avro/avro.go | 8 +- pkg/sink/codec/avro/avro_test.go | 6 +- pkg/sink/codec/avro/decoder.go | 32 +- pkg/sink/codec/avro/decoder_test.go | 15 +- pkg/sink/codec/canal/canal_encoder_test.go | 2 +- pkg/sink/codec/canal/canal_entry.go | 6 +- pkg/sink/codec/canal/canal_entry_test.go | 4 +- pkg/sink/codec/canal/canal_json_message.go | 4 +- .../canal/canal_json_row_event_encoder_test.go | 2 +- .../canal/canal_json_txn_event_encoder_test.go | 2 +- pkg/sink/codec/canal/canal_test_util.go | 4 +- pkg/sink/codec/common/message_test.go | 6 +- pkg/sink/codec/common/verify_checksum.go | 190 +++ pkg/sink/codec/craft/craft_encoder_test.go | 2 +- pkg/sink/codec/craft/message_decoder.go | 4 +- pkg/sink/codec/craft/message_encoder.go | 2 +- pkg/sink/codec/csv/csv_decoder.go | 6 +- pkg/sink/codec/csv/csv_decoder_test.go | 6 +- pkg/sink/codec/csv/csv_encoder_test.go | 6 +- pkg/sink/codec/csv/csv_message.go | 10 +- pkg/sink/codec/csv/csv_message_test.go | 10 +- pkg/sink/codec/internal/batch_tester.go | 2 +- pkg/sink/codec/internal/column.go | 2 +- pkg/sink/codec/internal/java.go | 2 +- pkg/sink/codec/maxwell/maxwell_encoder_test.go | 4 +- pkg/sink/codec/maxwell/maxwell_message.go | 4 +- pkg/sink/codec/maxwell/maxwell_message_test.go | 2 +- pkg/sink/codec/open/open_protocol_decoder.go | 4 +- pkg/sink/codec/open/open_protocol_encoder_test.go | 4 +- pkg/sink/codec/open/open_protocol_message.go | 2 +- pkg/sink/codec/open/open_protocol_message_test.go | 2 +- pkg/sink/kafka/claimcheck/claim_check_test.go | 2 +- pkg/sink/mysql/db_helper.go | 4 +- pkg/spanz/span.go | 8 +- pkg/spanz/span_test.go | 2 +- pkg/sqlmodel/causality.go | 8 +- pkg/sqlmodel/row_change.go | 4 +- pkg/sqlmodel/row_change_test.go | 13 +- pkg/sqlmodel/utils.go | 2 +- pkg/sqlmodel/where_handle.go | 6 +- pkg/sqlmodel/where_handle_test.go | 6 +- pkg/txnutil/gc/gc_manager.go | 8 +- pkg/txnutil/gc/gc_manager_test.go | 2 +- pkg/upstream/upstream.go | 2 +- pkg/util/memory.go | 2 +- pkg/util/tz.go | 31 +- pkg/version/check.go | 2 +- pkg/version/check_test.go | 2 +- tests/integration_tests/cdc/dailytest/db.go | 2 +- tests/integration_tests/cdc/dailytest/job.go | 4 +- tests/integration_tests/cdc/dailytest/parser.go | 8 +- tests/integration_tests/resolve_lock/main.go | 8 +- tests/integration_tests/util/db.go | 2 +- 305 files changed, 2043 insertions(+), 2191 deletions(-) create mode 100644 pkg/sink/codec/common/verify_checksum.go ๐Ÿงพ Pre-merged result: 10f68e69e5863d5c0b43c90372b2921ea2b0650f 10f68e69e Merge commit 'aa323aa99660421afc797ea115dbb30242e573cf' into HEAD aa323aa99 fix potential typos. 13c5e9410 update โœ… Pre merged ๐ŸŽ‰ โœ… ~~~~~All done.~~~~~~ [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // retry [Pipeline] } Cache not saved (git/pingcap/tiflow/rev-823a389-aa323aa already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (prepare) [Pipeline] timeout Timeout set to expire in 20 min [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/third_party_download [Pipeline] { [Pipeline] retry [Pipeline] { [Pipeline] sh + cd ../tiflow + ./scripts/download-integration-test-binaries.sh release-7.5 Download binaries... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 139 0 --:--:-- --:--:-- --:--:-- 138 100 41 100 41 0 0 139 0 --:--:-- --:--:-- --:--:-- 138 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 206 0 --:--:-- --:--:-- --:--:-- 207 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 194 0 --:--:-- --:--:-- --:--:-- 195 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 41 100 41 0 0 857 0 --:--:-- --:--:-- --:--:-- 872 >>> download tidb-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tidb/e406d5780b18a1f2aaf6230cde4b6403991e228d/centos7/tidb-server.tar.gz 2024-04-26 18:56:53 URL:http://fileserver.pingcap.net/download/builds/pingcap/tidb/e406d5780b18a1f2aaf6230cde4b6403991e228d/centos7/tidb-server.tar.gz [511580034/511580034] -> "tmp/tidb-server.tar.gz" [1] >>> download pd-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/pd/58453365285465cd90bc4472cff2bad7ce4d764b/centos7/pd-server.tar.gz 2024-04-26 18:57:07 URL:http://fileserver.pingcap.net/download/builds/pingcap/pd/58453365285465cd90bc4472cff2bad7ce4d764b/centos7/pd-server.tar.gz [173219013/173219013] -> "tmp/pd-server.tar.gz" [1] >>> download tikv-server.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tikv/46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990/centos7/tikv-server.tar.gz 2024-04-26 18:57:39 URL:http://fileserver.pingcap.net/download/builds/pingcap/tikv/46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990/centos7/tikv-server.tar.gz [810252984/810252984] -> "tmp/tikv-server.tar.gz" [1] >>> download tiflash.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-7.5/9002cc34d3b593a718b6c5260ba18f30a45ab314/centos7/tiflash.tar.gz 2024-04-26 18:57:56 URL:http://fileserver.pingcap.net/download/builds/pingcap/tiflash/release-7.5/9002cc34d3b593a718b6c5260ba18f30a45ab314/centos7/tiflash.tar.gz [263167205/263167205] -> "tmp/tiflash.tar.gz" [1] >>> download minio.tar.gz from http://fileserver.pingcap.net/download/minio.tar.gz 2024-04-26 18:58:00 URL:http://fileserver.pingcap.net/download/minio.tar.gz [17718777/17718777] -> "tmp/minio.tar.gz" [1] >>> download go-ycsb from http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb 2024-04-26 18:58:02 URL:http://fileserver.pingcap.net/download/builds/pingcap/go-ycsb/test-br/go-ycsb [45975512/45975512] -> "third_bin/go-ycsb" [1] >>> download jq from http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 2024-04-26 18:58:02 URL:http://fileserver.pingcap.net/download/builds/pingcap/test/jq-1.6/jq-linux64 [3953824/3953824] -> "third_bin/jq" [1] >>> download etcd.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz 2024-04-26 18:58:02 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/etcd-v3.4.7-linux-amd64.tar.gz [17310840/17310840] -> "tmp/etcd.tar.gz" [1] >>> download sync_diff_inspector.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-d671b084_linux-amd64.tar.gz 2024-04-26 18:58:04 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/sync_diff_inspector_hash-d671b084_linux-amd64.tar.gz [79877126/79877126] -> "tmp/sync_diff_inspector.tar.gz" [1] >>> download schema-registry.tar.gz from http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz 2024-04-26 18:58:13 URL:http://fileserver.pingcap.net/download/builds/pingcap/cdc/schema-registry.tar.gz [278386006/278386006] -> "tmp/schema-registry.tar.gz" [1] Download SUCCESS + ls -alh ./bin total 1.7G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 26 18:58 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 26 18:58 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 26 18:58 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 26 18:58 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 18 15:55 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 01:00 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 18 15:55 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 01:00 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 18 15:55 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 18 15:55 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M Apr 18 15:25 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M Apr 18 15:48 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 33M Apr 3 18:05 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 42M Apr 3 18:05 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M Apr 3 18:05 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 pd-recover -rwxr-xr-x 1 jenkins jenkins 102M Apr 3 18:04 pd-server -rwxr-xr-x 1 jenkins jenkins 25M Apr 3 18:05 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Sep 22 2023 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M Apr 22 17:56 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M Apr 18 15:56 tiflash -rwxr-xr-x 1 jenkins jenkins 353M Apr 25 18:18 tikv-server + make check_third_party_binary /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/tidb-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/tikv-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/pd-server /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/tiflash /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/pd-ctl /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/sync_diff_inspector /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/go-ycsb /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/etcdctl /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/jq /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/minio /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/bin/bin/schema-registry-start + cd - /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/third_party_download + mkdir -p bin + mv ../tiflow/bin/bin ../tiflow/bin/etc ../tiflow/bin/etcdctl ../tiflow/bin/go-ycsb ../tiflow/bin/jq ../tiflow/bin/lib ../tiflow/bin/libc++.so.1 ../tiflow/bin/libc++.so.1.0 ../tiflow/bin/libc++abi.so.1 ../tiflow/bin/libc++abi.so.1.0 ../tiflow/bin/libgmssl.so ../tiflow/bin/libgmssl.so.3 ../tiflow/bin/libgmssl.so.3.0 ../tiflow/bin/libtiflash_proxy.so ../tiflow/bin/minio ../tiflow/bin/pd-api-bench ../tiflow/bin/pd-ctl ../tiflow/bin/pd-heartbeat-bench ../tiflow/bin/pd-recover ../tiflow/bin/pd-server ../tiflow/bin/pd-tso-bench ../tiflow/bin/regions-dump ../tiflow/bin/share ../tiflow/bin/stores-dump ../tiflow/bin/sync_diff_inspector ../tiflow/bin/tidb-server ../tiflow/bin/tiflash ../tiflow/bin/tikv-server ./bin/ + ls -alh ./bin total 1.7G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 26 18:58 . drwxr-sr-x 3 jenkins jenkins 4.0K Apr 26 18:58 .. drwxr-sr-x 2 jenkins jenkins 4.0K May 19 2023 bin drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 2 2020 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 26 18:58 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 26 18:58 jq drwxr-sr-x 3 jenkins jenkins 4.0K May 10 2023 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 18 15:55 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Nov 7 01:00 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 18 15:55 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Nov 7 01:00 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 18 15:55 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 18 15:55 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M Apr 18 15:25 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M Apr 18 15:48 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Jul 29 2020 minio -rwxr-xr-x 1 jenkins jenkins 33M Apr 3 18:05 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 42M Apr 3 18:05 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M Apr 3 18:05 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 pd-recover -rwxr-xr-x 1 jenkins jenkins 102M Apr 3 18:04 pd-server -rwxr-xr-x 1 jenkins jenkins 25M Apr 3 18:05 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K May 10 2023 share -rwxr-xr-x 1 jenkins jenkins 29M Apr 3 18:05 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Sep 22 2023 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M Apr 22 17:56 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M Apr 18 15:56 tiflash -rwxr-xr-x 1 jenkins jenkins 353M Apr 25 18:18 tikv-server + ./bin/tidb-server -V Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore + ./bin/pd-server -V Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 + ./bin/tikv-server -V TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + ./bin/tiflash --version TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure + ./bin/sync_diff_inspector --version App Name: sync_diff_inspector v2.0 Release Version: v7.4.0 Git Commit Hash: d671b0840063bc2532941f02e02e12627402844c Git Branch: heads/refs/tags/v7.4.0 UTC Build Time: 2023-09-22 03:51:56 Go Version: go1.21.1 [Pipeline] } [Pipeline] // retry [Pipeline] } [Pipeline] // dir [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + ls -alh ./bin total 8.0K drwxr-sr-x 2 jenkins jenkins 4.0K Apr 26 18:58 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 26 18:58 .. + '[' -f ./bin/cdc ']' + make cdc CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g10f68e69e" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 10:58:16" -X "github.com/pingcap/tiflow/pkg/version.GitHash=10f68e69e5863d5c0b43c90372b2921ea2b0650f" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g10f68e69e"' -o bin/cdc ./cmd/cdc/main.go go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/gin-gonic/gin v1.8.1 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading github.com/google/btree v1.1.2 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading golang.org/x/net v0.24.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading github.com/robfig/cron v1.2.0 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading cloud.google.com/go v0.112.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/r3labs/diff v1.1.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/google/uuid v1.6.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14 go: downloading github.com/swaggo/gin-swagger v1.2.0 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/stretchr/testify v1.9.0 go: downloading blainsmith.com/go/seahash v1.2.1 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/goccy/go-json v0.10.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/klauspost/compress v1.17.7 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading golang.org/x/sys v0.19.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20230619063055-fd67d94318fd go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/go-playground/validator/v10 v10.11.1 go: downloading github.com/pelletier/go-toml/v2 v2.0.5 go: downloading github.com/ugorji/go/codec v1.2.7 go: downloading google.golang.org/protobuf v1.33.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/golang/protobuf v1.5.4 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/golang/mock v1.6.0 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading cloud.google.com/go/compute v1.24.0 go: downloading cloud.google.com/go/iam v1.1.6 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading google.golang.org/api v0.162.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/swaggo/swag v1.8.3 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading golang.org/x/text v0.14.0 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.8 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/go-playground/universal-translator v0.18.0 go: downloading github.com/leodido/go-urn v1.2.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.17.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/pingcap/fn v1.0.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/go-playground/locales v0.14.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/ardielle/ardielle-go v1.5.2 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/KyleBanks/depth v1.2.1 go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading github.com/go-openapi/spec v0.20.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/go-openapi/jsonpointer v0.19.6 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/rivo/uniseg v0.4.6 go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading k8s.io/apimachinery v0.28.6 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/felixge/httpsnoop v1.0.4 + '[' -f ./bin/cdc_kafka_consumer ']' + make kafka_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g10f68e69e" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:20" -X "github.com/pingcap/tiflow/pkg/version.GitHash=10f68e69e5863d5c0b43c90372b2921ea2b0650f" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g10f68e69e"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go + '[' -f ./bin/cdc_storage_consumer ']' + make storage_consumer CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g10f68e69e" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:27" -X "github.com/pingcap/tiflow/pkg/version.GitHash=10f68e69e5863d5c0b43c90372b2921ea2b0650f" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g10f68e69e"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go + '[' -f ./bin/cdc.test ']' + make integration_test_build cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd go: downloading github.com/sergi/go-diff v1.1.0 CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g10f68e69e" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:36" -X "github.com/pingcap/tiflow/pkg/version.GitHash=10f68e69e5863d5c0b43c90372b2921ea2b0650f" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g10f68e69e"' -o bin/cdc_storage_consumer ./cmd/storage-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g10f68e69e" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:36" -X "github.com/pingcap/tiflow/pkg/version.GitHash=10f68e69e5863d5c0b43c90372b2921ea2b0650f" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g10f68e69e"' -o bin/cdc_kafka_consumer ./cmd/kafka-consumer/main.go CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g10f68e69e" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:36" -X "github.com/pingcap/tiflow/pkg/version.GitHash=10f68e69e5863d5c0b43c90372b2921ea2b0650f" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g10f68e69e"' -o bin/cdc_pulsar_consumer ./cmd/pulsar-consumer/main.go $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null) go: downloading gorm.io/gorm v1.23.8 go: downloading github.com/PingCAP-QE/go-sqlsmith v0.0.0-20231213065948-336e064b488d go: downloading github.com/chzyer/readline v1.5.1 go: downloading github.com/deepmap/oapi-codegen v1.9.0 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/getkin/kin-openapi v0.80.0 go: downloading github.com/gogo/gateway v1.1.0 go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954 go: downloading go.uber.org/dig v1.13.0 go: downloading github.com/mattn/go-shellwords v1.0.12 go: downloading go.uber.org/ratelimit v0.2.0 go: downloading github.com/VividCortex/mysqlerr v1.0.0 go: downloading github.com/glebarez/go-sqlite v1.17.3 go: downloading github.com/go-ozzo/ozzo-validation/v4 v4.3.0 go: downloading github.com/glebarez/sqlite v1.4.6 go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17 go: downloading gorm.io/driver/mysql v1.3.3 go: downloading go.uber.org/goleak v1.3.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc go: downloading github.com/jmoiron/sqlx v1.3.3 go: downloading upper.io/db.v3 v3.7.1+incompatible go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/shurcooL/httpgzip v0.0.0-20190720172056-320755c1c1b0 go: downloading github.com/andres-erbsen/clock v0.0.0-20160526145045-9e14626cd129 go: downloading modernc.org/libc v1.16.8 go: downloading modernc.org/sqlite v1.17.3 go: downloading github.com/ngaut/log v0.0.0-20210830112240-0124ec040aeb go: downloading github.com/jinzhu/inflection v1.0.0 go: downloading github.com/jinzhu/now v1.1.5 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/ghodss/yaml v1.0.0 go: downloading github.com/rs/cors v1.7.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading modernc.org/memory v1.1.1 go: downloading modernc.org/mathutil v1.6.0 CGO_ENABLED=1 GO111MODULE=on go test -p 3 --race --tags=intest -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g10f68e69e" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:36" -X "github.com/pingcap/tiflow/pkg/version.GitHash=10f68e69e5863d5c0b43c90372b2921ea2b0650f" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g10f68e69e"' -c -cover -covermode=atomic \ -coverpkg=github.com/pingcap/tiflow/... \ -o bin/cdc.test github.com/pingcap/tiflow/cmd/cdc \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } CGO_ENABLED=0 GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v7.5.1-57-g10f68e69e" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-04-26 11:01:36" -X "github.com/pingcap/tiflow/pkg/version.GitHash=10f68e69e5863d5c0b43c90372b2921ea2b0650f" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=HEAD" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 linux/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v7.5.1-57-g10f68e69e"' -o bin/cdc ./cmd/cdc/main.go \ || { $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null); exit 1; } $(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl disable >/dev/null) + ls -alh ./bin total 1.1G drwxr-sr-x 2 jenkins jenkins 4.0K Apr 26 19:06 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 26 18:58 .. -rwxr-xr-x 1 jenkins jenkins 203M Apr 26 19:06 cdc -rwxr-xr-x 1 jenkins jenkins 338M Apr 26 19:06 cdc.test -rwxr-xr-x 1 jenkins jenkins 182M Apr 26 19:01 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 182M Apr 26 19:02 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 181M Apr 26 19:01 cdc_storage_consumer + ./bin/cdc version Release Version: v7.5.1-57-g10f68e69e Git Commit Hash: 10f68e69e5863d5c0b43c90372b2921ea2b0650f Git Branch: HEAD UTC Build Time: 2024-04-26 11:01:36 Go Version: go version go1.21.6 linux/amd64 Failpoint Build: true [Pipeline] } Cache saved successfully (binary/pingcap/tiflow/cdc-integration-test/rev-823a389-aa323aa) 1136782848 bytes in 20.55 secs (55322606 bytes/sec) [Pipeline] // cache [Pipeline] cache Cache not restored (no such key found) [Pipeline] { [Pipeline] sh + cp -r ../third_party_download/bin/bin ../third_party_download/bin/etc ../third_party_download/bin/etcdctl ../third_party_download/bin/go-ycsb ../third_party_download/bin/jq ../third_party_download/bin/lib ../third_party_download/bin/libc++.so.1 ../third_party_download/bin/libc++.so.1.0 ../third_party_download/bin/libc++abi.so.1 ../third_party_download/bin/libc++abi.so.1.0 ../third_party_download/bin/libgmssl.so ../third_party_download/bin/libgmssl.so.3 ../third_party_download/bin/libgmssl.so.3.0 ../third_party_download/bin/libtiflash_proxy.so ../third_party_download/bin/minio ../third_party_download/bin/pd-api-bench ../third_party_download/bin/pd-ctl ../third_party_download/bin/pd-heartbeat-bench ../third_party_download/bin/pd-recover ../third_party_download/bin/pd-server ../third_party_download/bin/pd-tso-bench ../third_party_download/bin/regions-dump ../third_party_download/bin/share ../third_party_download/bin/stores-dump ../third_party_download/bin/sync_diff_inspector ../third_party_download/bin/tidb-server ../third_party_download/bin/tiflash ../third_party_download/bin/tikv-server ./bin/ + ls -alh ./bin total 2.7G drwxr-sr-x 6 jenkins jenkins 4.0K Apr 26 19:07 . drwxr-sr-x 19 jenkins jenkins 4.0K Apr 26 18:58 .. drwxr-sr-x 2 jenkins jenkins 4.0K Apr 26 19:07 bin -rwxr-xr-x 1 jenkins jenkins 203M Apr 26 19:06 cdc -rwxr-xr-x 1 jenkins jenkins 338M Apr 26 19:06 cdc.test -rwxr-xr-x 1 jenkins jenkins 182M Apr 26 19:01 cdc_kafka_consumer -rwxr-xr-x 1 jenkins jenkins 182M Apr 26 19:02 cdc_pulsar_consumer -rwxr-xr-x 1 jenkins jenkins 181M Apr 26 19:01 cdc_storage_consumer drwxr-sr-x 4 jenkins jenkins 4.0K Apr 26 19:07 etc -rwxr-xr-x 1 jenkins jenkins 17M Apr 26 19:07 etcdctl -rwxr-xr-x 1 jenkins jenkins 44M Apr 26 19:07 go-ycsb -rwxr-xr-x 1 jenkins jenkins 3.8M Apr 26 19:07 jq drwxr-sr-x 3 jenkins jenkins 4.0K Apr 26 19:07 lib lrwxrwxrwx 1 jenkins jenkins 13 Apr 26 19:07 libc++.so.1 -> libc++.so.1.0 -rwxr-xr-x 1 jenkins jenkins 1016K Apr 26 19:07 libc++.so.1.0 lrwxrwxrwx 1 jenkins jenkins 16 Apr 26 19:07 libc++abi.so.1 -> libc++abi.so.1.0 -rwxr-xr-x 1 jenkins jenkins 358K Apr 26 19:07 libc++abi.so.1.0 lrwxrwxrwx 1 jenkins jenkins 13 Apr 26 19:07 libgmssl.so -> libgmssl.so.3 lrwxrwxrwx 1 jenkins jenkins 15 Apr 26 19:07 libgmssl.so.3 -> libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 2.6M Apr 26 19:07 libgmssl.so.3.0 -rwxr-xr-x 1 jenkins jenkins 88M Apr 26 19:07 libtiflash_proxy.so -rwxr-xr-x 1 jenkins jenkins 50M Apr 26 19:07 minio -rwxr-xr-x 1 jenkins jenkins 33M Apr 26 19:07 pd-api-bench -rwxr-xr-x 1 jenkins jenkins 42M Apr 26 19:07 pd-ctl -rwxr-xr-x 1 jenkins jenkins 30M Apr 26 19:07 pd-heartbeat-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 26 19:07 pd-recover -rwxr-xr-x 1 jenkins jenkins 102M Apr 26 19:07 pd-server -rwxr-xr-x 1 jenkins jenkins 25M Apr 26 19:07 pd-tso-bench -rwxr-xr-x 1 jenkins jenkins 29M Apr 26 19:07 regions-dump drwxr-sr-x 4 jenkins jenkins 4.0K Apr 26 19:07 share -rwxr-xr-x 1 jenkins jenkins 29M Apr 26 19:07 stores-dump -rwxr-xr-x 1 jenkins jenkins 192M Apr 26 19:07 sync_diff_inspector -rwxr-xr-x 1 jenkins jenkins 199M Apr 26 19:07 tidb-server -rwxr-xr-x 1 jenkins jenkins 377M Apr 26 19:07 tiflash -rwxr-xr-x 1 jenkins jenkins 353M Apr 26 19:07 tikv-server [Pipeline] } Cache saved successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 74.94 secs (44903540 bytes/sec) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Tests) [Pipeline] parallel [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G00') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G01') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G02') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G03') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G04') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G05') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G06') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G07') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G08') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G09') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G10') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G11') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G12') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G13') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G14') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G15') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G16') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G17') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G18') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G19') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G20') [Pipeline] { (Branch: Matrix - TEST_GROUP = 'G21') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G00') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G01') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G02') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G03') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G04') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G05') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G06') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G07') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G08') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G09') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G10') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G11') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G12') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G13') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G14') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G15') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G16') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G17') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G18') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G19') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G20') [Pipeline] stage [Pipeline] { (Matrix - TEST_GROUP = 'G21') [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted [Pipeline] readTrusted Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh [Pipeline] node Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-p6lhp-w226n --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "bf823cece05b181930af9a10ea4815c1e35a925c" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-p6lhp" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d [Pipeline] node Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-qs16h-gm1sf --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "d4955302a7871c431a9b6c70d937fd846aab00df" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-qs16h" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { [Pipeline] checkout The recommended git tool is: git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf [Pipeline] node No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6c7a0d0b; decorates RemoteLauncher[hudson.remoting.Channel@43d4a23f:JNLP4-connect connection from 10.233.105.115/10.233.105.115:41282] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3091a2c1; decorates RemoteLauncher[hudson.remoting.Channel@47ce4faf:JNLP4-connect connection from 10.233.127.34/10.233.127.34:58134] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { [Pipeline] node Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git [Pipeline] podTemplate [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk [Pipeline] node > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-zhb19-7d0px --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "3094fdeca703daf857eb88560f63942d031eeb82" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-zhb19" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Commit message: "feat(tidb): test flashbacktest package (#2942)" [Pipeline] withEnv [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] withEnv Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d [Pipeline] container [Pipeline] { [Pipeline] { Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh [Pipeline] podTemplate [Pipeline] { [Pipeline] checkout [Pipeline] node [Pipeline] stage [Pipeline] { (Test) The recommended git tool is: git [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 [Pipeline] { [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 [Pipeline] cache Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-8jzdv-wp2js --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "e13c6a1fbb603693701c7723572a5e88c1b5441f" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-8jzdv" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-4qcfc-p3ms2 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c42bc9e2b2aad5d9ec2907ca6095700347d38e9f" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-4qcfc" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@180be2c2; decorates RemoteLauncher[hudson.remoting.Channel@51d658c7:JNLP4-connect connection from 10.233.106.65/10.233.106.65:45172] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-nnw0d-cvh2w --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "c5ee6fcfeef672a5243ef4f064f0ac69b5bd4edf" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-nnw0d" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-rv597-fqv7l --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "dd04a2adf4434a2f82e86f6a0d1e9442734f1dda" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-rv597" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-fz0c2-f2fw7 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "6bc05dc44a7cdd06b6aa55bd83989d706f2643c5" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-fz0c2" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-7rhcc-c7p0r --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "28eb223c30f2b636f9efe4bbc83eeef7ff225c15" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-7rhcc" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-1p0j0-p472d --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "194e0a80828795f28a5b5bf68eb2f19fe1695e1d" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-1p0j0" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-4gkcs-d1jqx --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "4ecb8614dc5acd3f6661fae88f00a7dd31f68d64" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-4gkcs" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 6.28 secs (536061984 bytes/sec) [Pipeline] { [Pipeline] cache Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp Obtained pipelines/pingcap/tiflow/release-7.5/pod-pull_cdc_integration_mysql_test.yaml from git https://github.com/PingCAP-QE/ci.git Still waiting to schedule task โ€˜ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjpโ€™ is offline Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-hz3f1-b5zq0 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ea977180157976963cb6adbedc43cce1c0011cc6" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-hz3f1" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 6.77 secs (496913919 bytes/sec) [Pipeline] { [Pipeline] sh [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G00 Run cases: bdr_mode capture_suicide_while_balance_table syncpoint hang_sink_suicide server_config_compatibility kafka_big_messages kafka_compression kafka_messages kafka_sink_error_resume mq_sink_lost_callback mq_sink_dispatcher kafka_column_selector kafka_column_selector_avro lossy_ddl storage_csv_update PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=94ae9324-a679-4fa9-be83-16d9709d18b2 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G00 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-p6lhp GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73 ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-p6lhp GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/bdr_mode/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G01 Run cases: http_api http_api_tls api_v2 canal_json_adapter_compatibility canal_json_basic multi_topics avro_basic canal_json_handle_key_only open_protocol_handle_key_only canal_json_claim_check open_protocol_claim_check canal_json_storage_basic canal_json_storage_partition_table multi_tables_ddl PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=f17eb605-1c43-41a1-b083-4c6ff60d5add BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G01 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-qs16h GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-qs16h GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_api/run.sh using Sink-Type: mysql... <<================= [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { [Pipeline] podTemplate [Pipeline] { WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. [Pipeline] { [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node Collecting requests==2.26.0 [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] node [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { Downloading https://files.pythonhosted.org/packages/92/96/144f70b972a9c0eabbd4391ef93ccd49d0f2747f4f6a2a2738e99e5adc65/requests-2.26.0-py2.py3-none-any.whl (62kB) Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0 Collecting urllib3<1.27,>=1.21.1 (from requests==2.26.0) Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7 Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b Downloading https://files.pythonhosted.org/packages/b0/53/aa91e163dcfd1e5b82d8a890ecf13314e3e149c05270cc644581f77f17fd/urllib3-1.26.18-py2.py3-none-any.whl (143kB) Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g [Pipeline] { Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr [Pipeline] container [Pipeline] { [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] checkout Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Collecting idna<4,>=2.5; python_version >= "3" (from requests==2.26.0) The recommended git tool is: git Downloading https://files.pythonhosted.org/packages/e5/3e/741d8c82801c347547f8a2a06aa57dbb1992be9e948df2ea0eda2c8b79e8/idna-3.7-py3-none-any.whl (66kB) start tidb cluster in /tmp/tidb_cdc_test/bdr_mode Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { Collecting certifi>=2017.4.17 (from requests==2.26.0) Downloading https://files.pythonhosted.org/packages/ba/06/a07f096c664aeb9f01624f858c3add0a4e913d6c96257acb4fce61e7de14/certifi-2024.2.2-py3-none-any.whl (163kB) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@69aa7184; decorates RemoteLauncher[hudson.remoting.Channel@777d3414:JNLP4-connect connection from 10.233.100.95/10.233.100.95:43826] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4b54da20; decorates RemoteLauncher[hudson.remoting.Channel@5538b64f:JNLP4-connect connection from 10.233.97.28/10.233.97.28:51856] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 Installing collected packages: urllib3, idna, certifi, requests Found existing installation: urllib3 1.26.16 Uninstalling urllib3-1.26.16: Successfully uninstalled urllib3-1.26.16 Found existing installation: idna 3.4 Uninstalling idna-3.4: Successfully uninstalled idna-3.4 Found existing installation: certifi 2023.5.7 Uninstalling certifi-2023.5.7: Successfully uninstalled certifi-2023.5.7 Found existing installation: requests 2.27.1 Uninstalling requests-2.27.1: Successfully uninstalled requests-2.27.1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@3c694f7b; decorates RemoteLauncher[hudson.remoting.Channel@55e120fe:JNLP4-connect connection from 10.233.84.95/10.233.84.95:35024] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@55af6357; decorates RemoteLauncher[hudson.remoting.Channel@14bdd70c:JNLP4-connect connection from 10.233.67.192/10.233.67.192:34042] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4763499f; decorates RemoteLauncher[hudson.remoting.Channel@683f2df5:JNLP4-connect connection from 10.233.71.151/10.233.71.151:45806] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Successfully installed certifi-2024.2.2 idna-3.7 requests-2.26.0 urllib3-1.26.18 [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@14fed7fd; decorates RemoteLauncher[hudson.remoting.Channel@d56925f:JNLP4-connect connection from 10.233.88.141/10.233.88.141:51478] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@63a65e07; decorates RemoteLauncher[hudson.remoting.Channel@6a2a55c0:JNLP4-connect connection from 10.233.86.191/10.233.86.191:43212] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 The 1 times to try to start tidb cluster... [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7a6e1252; decorates RemoteLauncher[hudson.remoting.Channel@16a84768:JNLP4-connect connection from 10.233.90.136/10.233.90.136:55912] will be ignored (a typical symptom is the Git executable not being run inside a designated container) [Pipeline] dir Cloning the remote Git repository Using shallow clone with depth 1 Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] cache No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@12c5528f; decorates RemoteLauncher[hudson.remoting.Channel@64bc11ae:JNLP4-connect connection from 10.233.88.81/10.233.88.81:33724] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-cxw2c-5c8jp --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "9c049055825772d1641857d692c3d9ee95fbb7bc" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-cxw2c" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Commit message: "feat(tidb): test flashbacktest package (#2942)" Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Commit message: "feat(tidb): test flashbacktest package (#2942)" Commit message: "feat(tidb): test flashbacktest package (#2942)" start tidb cluster in /tmp/tidb_cdc_test/http_api Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Commit message: "feat(tidb): test flashbacktest package (#2942)" Commit message: "feat(tidb): test flashbacktest package (#2942)" Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-0wbvc-g82vx --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "237afb8693b4fb995efebd8ac7cb8cdfcc9bfb9a" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-0wbvc" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-9hrjs-dws63 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "b08339bdc58924e043a8c09512ec88ceb6ed3839" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-9hrjs" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Starting Upstream TiDB... Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Verifying downstream PD is started... Created Pod: kubernetes jenkins-tiflow/ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-vzs14-j9lbm --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "3328451555cae58cc3463400e988f2c36386cdff" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-vzs14" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-xm95t-s7pn5 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "20ac69776da4da25df4add2fac3e89a73eb38c32" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-xm95t" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Starting Upstream TiDB... Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Still waiting to schedule task โ€˜ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7nโ€™ is offline Still waiting to schedule task โ€˜ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4โ€™ is offline ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Still waiting to schedule task โ€˜ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1gโ€™ is offline Still waiting to schedule task โ€˜ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4gโ€™ is offline Still waiting to schedule task โ€˜ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzrโ€™ is offline Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-2jdqt-kxr51 --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "d8b741f670fcf0aae7888efaff2121dc4aaa2133" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-2jdqt" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6861857c000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:1171, start at 2024-04-26 19:09:16.52353625 +0800 CST m=+5.438516768 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:16.531 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:16.511 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:16.511 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6861857c000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:1171, start at 2024-04-26 19:09:16.52353625 +0800 CST m=+5.438516768 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:16.531 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:16.511 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:16.511 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686187140004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:1239, start at 2024-04-26 19:09:16.617373496 +0800 CST m=+5.472418483 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:16.625 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:16.613 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:16.613 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/bdr_mode/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/bdr_mode/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/bdr_mode/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-3dv99-2c31p --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "ce0ade621fd620dc5ecb3eb28f7b64bd235cfe5b" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-3dv99" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test [Fri Apr 26 19:09:21 CST 2024] <<<<<< START cdc server in bdr_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.25602563.out server --log-file /tmp/tidb_cdc_test/bdr_mode/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/bdr_mode/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-r4370-bp0zl --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "2d2423f758fc3631c9937c905ca27f363afa50d2" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-r4370" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:09:24 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e141718-1334-4b16-b818-d9d05eb22f6e {"id":"2e141718-1334-4b16-b818-d9d05eb22f6e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1832ddf2 2e141718-1334-4b16-b818-d9d05eb22f6e /tidb/cdc/default/default/upstream/7362131206873163221 {"id":7362131206873163221,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e141718-1334-4b16-b818-d9d05eb22f6e {"id":"2e141718-1334-4b16-b818-d9d05eb22f6e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1832ddf2 2e141718-1334-4b16-b818-d9d05eb22f6e /tidb/cdc/default/default/upstream/7362131206873163221 {"id":7362131206873163221,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e141718-1334-4b16-b818-d9d05eb22f6e {"id":"2e141718-1334-4b16-b818-d9d05eb22f6e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1832ddf2 2e141718-1334-4b16-b818-d9d05eb22f6e /tidb/cdc/default/default/upstream/7362131206873163221 {"id":7362131206873163221,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:09:24 CST 2024] <<<<<< START cdc server in bdr_mode case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.26082610.out server --log-file /tmp/tidb_cdc_test/bdr_mode/cdcdown.log --log-level debug --data-dir /tmp/tidb_cdc_test/bdr_mode/cdc_datadown --cluster-id default --addr 127.0.0.1:8400 --pd http://127.0.0.1:2479 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info * About to connect() to 127.0.0.1 port 8400 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8400; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6862127c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:1220, start at 2024-04-26 19:09:25.560055038 +0800 CST m=+10.744459200 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:25.570 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:25.535 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:25.535 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6862127c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:1220, start at 2024-04-26 19:09:25.560055038 +0800 CST m=+10.744459200 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:25.570 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:25.535 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:25.535 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686210b00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:1292, start at 2024-04-26 19:09:25.444148939 +0800 CST m=+10.568384338 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:25.453 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:25.420 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:25.420 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/http_api/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/http_api/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8400/debug/info * About to connect() to 127.0.0.1 port 8400 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8400 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8400 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:09:27 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/168f38af-39f4-45be-8781-53db019ed1a9 {"id":"168f38af-39f4-45be-8781-53db019ed1a9","address":"127.0.0.1:8400","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/35118f1a18331023 168f38af-39f4-45be-8781-53db019ed1a9 /tidb/cdc/default/default/upstream/7362131196478202750 {"id":7362131196478202750,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/168f38af-39f4-45be-8781-53db019ed1a9 {"id":"168f38af-39f4-45be-8781-53db019ed1a9","address":"127.0.0.1:8400","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/35118f1a18331023 168f38af-39f4-45be-8781-53db019ed1a9 /tidb/cdc/default/default/upstream/7362131196478202750 {"id":7362131196478202750,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/168f38af-39f4-45be-8781-53db019ed1a9 {"id":"168f38af-39f4-45be-8781-53db019ed1a9","address":"127.0.0.1:8400","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/35118f1a18331023 168f38af-39f4-45be-8781-53db019ed1a9 /tidb/cdc/default/default/upstream/7362131196478202750 {"id":7362131196478202750,"pd-endpoints":"http://127.0.0.1:2479,http://127.0.0.1:2479","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:09:29 CST 2024] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.26192621.out server --log-file /tmp/tidb_cdc_test/http_api/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table bdr_mode.t1 exists table bdr_mode.t1 exists + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.cli.2684.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306 -c test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/bdr_mode/conf/up.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362131206873163221,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306","create_time":"2024-04-26T19:09:29.991036869+08:00","start_ts":449348834377334790,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":true,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348834377334790,"checkpoint_ts":449348834377334790,"checkpoint_time":"2024-04-26 19:09:29.811"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4 is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-svhtl-wg40l --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "17ef20ef6159f60c77695abf2cd59c069570be24" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-svhtl" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4 in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 20.97 secs (160473537 bytes/sec) + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bdr_mode.cli.2717.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:4000 -c test-2 --server http://127.0.0.1:8400 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/bdr_mode/conf/down.toml [Pipeline] { + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > Create changefeed successfully! ID: test-2 Info: {"upstream_id":7362131196478202750,"namespace":"default","id":"test-2","sink_uri":"mysql://root@127.0.0.1:4000","create_time":"2024-04-26T19:09:31.924165822+08:00","start_ts":449348834902147075,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":true,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348834902147075,"checkpoint_ts":449348834902147075,"checkpoint_time":"2024-04-26 19:09:31.813"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:09:32 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aefada55-50d6-4a69-800a-c73186e1b945 {"id":"aefada55-50d6-4a69-800a-c73186e1b945","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b353 aefada55-50d6-4a69-800a-c73186e1b945 /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aefada55-50d6-4a69-800a-c73186e1b945 {"id":"aefada55-50d6-4a69-800a-c73186e1b945","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b353 aefada55-50d6-4a69-800a-c73186e1b945 /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aefada55-50d6-4a69-800a-c73186e1b945 {"id":"aefada55-50d6-4a69-800a-c73186e1b945","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b353 aefada55-50d6-4a69-800a-c73186e1b945 /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [Pipeline] { [Pipeline] sh + set +x "is-owner": true, run task successfully [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] checkout [Pipeline] container [Pipeline] { The recommended git tool is: git [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G03 Run cases: row_format drop_many_tables processor_stop_delay partition_table PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=bed388cf-84d4-41cc-8c6a-8b22699deb64 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G03 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-zhb19 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-zhb19 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/row_format/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] container owner pid: 2624 owner id aefada55-50d6-4a69-800a-c73186e1b945 default [Fri Apr 26 19:09:35 CST 2024] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.27232725.out server --log-file /tmp/tidb_cdc_test/http_api/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@4ee14aa; decorates RemoteLauncher[hudson.remoting.Channel@5b13321f:JNLP4-connect connection from 10.233.70.110/10.233.70.110:44320] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { [Pipeline] { table tidb_cdc.syncpoint_v1 exists table tidb_cdc.syncpoint_v1 does not exists table bdr_mode.finish_mark exists table bdr_mode.finish_mark exists check diff failed 1-th time, retry later [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git [Pipeline] checkout The recommended git tool is: git The recommended git tool is: git No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7c9f0763; decorates RemoteLauncher[hudson.remoting.Channel@3ed874ee:JNLP4-connect connection from 10.233.86.93/10.233.86.93:34968] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] checkout [Pipeline] checkout No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7ddb2c4; decorates RemoteLauncher[hudson.remoting.Channel@3a1957e:JNLP4-connect connection from 10.233.127.54/10.233.127.54:52974] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 The recommended git tool is: git [Pipeline] stage The recommended git tool is: git [Pipeline] { (Test) No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@1d12026f; decorates RemoteLauncher[hudson.remoting.Channel@5f42cbc3:JNLP4-connect connection from 10.233.123.163/10.233.123.163:32980] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@761d0c67; decorates RemoteLauncher[hudson.remoting.Channel@5f1ae1fd:JNLP4-connect connection from 10.233.123.213/10.233.123.213:57026] will be ignored (a typical symptom is the Git executable not being run inside a designated container) [Pipeline] { (Test)Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] stage Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { (Test) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] stage [Pipeline] { (Test) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@5d7ce72f; decorates RemoteLauncher[hudson.remoting.Channel@99d9030:JNLP4-connect connection from 10.233.88.7/10.233.88.7:43706] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN check diff successfully wait process cdc.test exit for 1-th time... [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@441476aa; decorates RemoteLauncher[hudson.remoting.Channel@74f5f68f:JNLP4-connect connection from 10.233.69.239/10.233.69.239:60904] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@76e18576; decorates RemoteLauncher[hudson.remoting.Channel@69a8a6ad:JNLP4-connect connection from 10.233.105.231/10.233.105.231:47356] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] { [Pipeline] { No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6d6a82ab; decorates RemoteLauncher[hudson.remoting.Channel@11fd7e76:JNLP4-connect connection from 10.233.105.169/10.233.105.169:47364] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 [Pipeline] { [Pipeline] { [Pipeline] { Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:09:38 GMT < Content-Length: 911 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8611bc78-b9e0-4b0b-836f-ef0915e74f1f {"id":"8611bc78-b9e0-4b0b-836f-ef0915e74f1f","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/aefada55-50d6-4a69-800a-c73186e1b945 {"id":"aefada55-50d6-4a69-800a-c73186e1b945","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b353 aefada55-50d6-4a69-800a-c73186e1b945 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b384 8611bc78-b9e0-4b0b-836f-ef0915e74f1f /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8611bc78-b9e0-4b0b-836f-ef0915e74f1f {"id":"8611bc78-b9e0-4b0b-836f-ef0915e74f1f","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/aefada55-50d6-4a69-800a-c73186e1b945 {"id":"aefada55-50d6-4a69-800a-c73186e1b945","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b353 aefada55-50d6-4a69-800a-c73186e1b945 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b384 8611bc78-b9e0-4b0b-836f-ef0915e74f1f /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8611bc78-b9e0-4b0b-836f-ef0915e74f1f {"id":"8611bc78-b9e0-4b0b-836f-ef0915e74f1f","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/aefada55-50d6-4a69-800a-c73186e1b945 {"id":"aefada55-50d6-4a69-800a-c73186e1b945","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b353 aefada55-50d6-4a69-800a-c73186e1b945 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b384 8611bc78-b9e0-4b0b-836f-ef0915e74f1f /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep -v "aefada55-50d6-4a69-800a-c73186e1b945 default" | grep id [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { wait process cdc.test exit for 2-th time... [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 wait process cdc.test exit for 3-th time... Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Commit message: "feat(tidb): test flashbacktest package (#2942)" "id": "8611bc78-b9e0-4b0b-836f-ef0915e74f1f", run task successfully > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 start tidb cluster in /tmp/tidb_cdc_test/row_format Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Commit message: "feat(tidb): test flashbacktest package (#2942)" capture_id: 8611bc78-b9e0-4b0b-836f-ef0915e74f1f Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) pass test: check health pass test: get status > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Commit message: "feat(tidb): test flashbacktest package (#2942)" > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 pass test: create changefeed Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Verifying downstream PD is started... Commit message: "feat(tidb): test flashbacktest package (#2942)" > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:09:42 CST 2024] <<<<<< run test case bdr_mode success! >>>>>> Commit message: "feat(tidb): test flashbacktest package (#2942)" Commit message: "feat(tidb): test flashbacktest package (#2942)" Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 table test.simple1 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.simple1 not exists for 2-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.simple1 exists pass test: list changefeed pass test: get changefeed ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: pause changefeed pass test: update changefeed wait process 2624 exit for 1-th time... wait process 2624 exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 2624 exit for 3-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (2624) - No such process wait process 2624 exit for 4-th time... process 2624 already exit cdc.test cli capture list --server http://127.0.0.1:8301 2>&1 |grep 8611bc78-b9e0-4b0b-836f-ef0915e74f1f -A1 | grep '"is-owner": true' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/capture_suicide_while_balance_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) "is-owner": true, run task successfully [Fri Apr 26 19:09:54 CST 2024] <<<<<< START cdc server in http_api case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api.29472949.out server --log-file /tmp/tidb_cdc_test/http_api/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api/cdc_data --cluster-id default * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 14.37 secs (234196989 bytes/sec) [Pipeline] { [Pipeline] cache start tidb cluster in /tmp/tidb_cdc_test/capture_suicide_while_balance_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:09:57 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/changefeed-test1 {UpstreamID:7362131218917024121 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:09:41.565597579 +0800 CST StartTs:449348837437341706 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037a2240 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348837450448903} {CheckpointTs:449348841225322506 MinTableBarrierTs:449348841487466508 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348841487466507, checkpointTs: 449348841487466507, state: Replicating changefeedID: default/changefeed-test3 {UpstreamID:7362131218917024121 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2024-04-26 19:09:41.782530652 +0800 CST StartTs:449348837503139847 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d96750 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348837503139854} {CheckpointTs:449348841225322506 MinTableBarrierTs:449348841487466508 AdminJobType:noop} span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348841487466507, checkpointTs: 449348841487466507, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3bd47665-7939-4586-9e52-fb9fe566e278 {"id":"3bd47665-7939-4586-9e52-fb9fe566e278","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8611bc78-b9e0-4b0b-836f-ef0915e74f1f {"id":"8611bc78-b9e0-4b0b-836f-ef0915e74f1f","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b384 8611bc78-b9e0-4b0b-836f-ef0915e74f1f /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b3ff 3bd47665-7939-4586-9e52-fb9fe566e278 /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:09:41.565597579+08:00","start-ts":449348837437341706,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348837450448903} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2024-04-26T19:09:41.680908532+08:00","start-ts":449348837476925447,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"stopped","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348839626244098} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2024-04-26T19:09:41.782530652+08:00","start-ts":449348837503139847,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348837503139854} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":449348841487466507,"min-table-barrier-ts":449348841487466508,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":449348837647056911,"min-table-barrier-ts":449348839390314506,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":449348841487466507,"min-table-barrier-ts":449348841487466508,"admin-job-type":0} /tidb/cdc/default/default/task/position/3bd47665-7939-4586-9e52-fb9fe566e278/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/3bd47665-7939-4586-9e52-fb9fe566e278/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8611bc78-b9e0-4b0b-836f-ef0915e74f1f/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8611bc78-b9e0-4b0b-836f-ef0915e74f1f/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/changefeed-test1 {UpstreamID:7362131218917024121 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:09:41.565597579 +0800 CST StartTs:449348837437341706 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037a2240 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348837450448903} {CheckpointTs:449348841225322506 MinTableBarrierTs:449348841487466508 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348841487466507, checkpointTs: 449348841487466507, state: Replicating changefeedID: default/changefeed-test3 {UpstreamID:7362131218917024121 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2024-04-26 19:09:41.782530652 +0800 CST StartTs:449348837503139847 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d96750 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348837503139854} {CheckpointTs:449348841225322506 MinTableBarrierTs:449348841487466508 AdminJobType:noop} span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348841487466507, checkpointTs: 449348841487466507, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3bd47665-7939-4586-9e52-fb9fe566e278 {"id":"3bd47665-7939-4586-9e52-fb9fe566e278","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8611bc78-b9e0-4b0b-836f-ef0915e74f1f {"id":"8611bc78-b9e0-4b0b-836f-ef0915e74f1f","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b384 8611bc78-b9e0-4b0b-836f-ef0915e74f1f /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b3ff 3bd47665-7939-4586-9e52-fb9fe566e278 /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:09:41.565597579+08:00","start-ts":449348837437341706,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348837450448903} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2024-04-26T19:09:41.680908532+08:00","start-ts":449348837476925447,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"stopped","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348839626244098} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2024-04-26T19:09:41.782530652+08:00","start-ts":449348837503139847,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348837503139854} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":449348841487466507,"min-table-barrier-ts":449348841487466508,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":449348837647056911,"min-table-barrier-ts":449348839390314506,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":449348841487466507,"min-table-barrier-ts":449348841487466508,"admin-job-type":0} /tidb/cdc/default/default/task/position/3bd47665-7939-4586-9e52-fb9fe566e278/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/3bd47665-7939-4586-9e52-fb9fe566e278/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8611bc78-b9e0-4b0b-836f-ef0915e74f1f/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8611bc78-b9e0-4b0b-836f-ef0915e74f1f/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/changefeed-test1 {UpstreamID:7362131218917024121 Namespace:default ID:changefeed-test1 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:09:41.565597579 +0800 CST StartTs:449348837437341706 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037a2240 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348837450448903} {CheckpointTs:449348841225322506 MinTableBarrierTs:449348841487466508 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348841487466507, checkpointTs: 449348841487466507, state: Replicating changefeedID: default/changefeed-test3 {UpstreamID:7362131218917024121 Namespace:default ID:changefeed-test3 SinkURI:blackhole:// CreateTime:2024-04-26 19:09:41.782530652 +0800 CST StartTs:449348837503139847 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003d96750 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348837503139854} {CheckpointTs:449348841225322506 MinTableBarrierTs:449348841487466508 AdminJobType:noop} span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348841487466507, checkpointTs: 449348841487466507, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3bd47665-7939-4586-9e52-fb9fe566e278 {"id":"3bd47665-7939-4586-9e52-fb9fe566e278","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8611bc78-b9e0-4b0b-836f-ef0915e74f1f {"id":"8611bc78-b9e0-4b0b-836f-ef0915e74f1f","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b384 8611bc78-b9e0-4b0b-836f-ef0915e74f1f /tidb/cdc/default/__cdc_meta__/owner/22318f1a1844b3ff 3bd47665-7939-4586-9e52-fb9fe566e278 /tidb/cdc/default/default/changefeed/info/changefeed-test1 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test1","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:09:41.565597579+08:00","start-ts":449348837437341706,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348837450448903} /tidb/cdc/default/default/changefeed/info/changefeed-test2 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test2","sink-uri":"blackhole://","create-time":"2024-04-26T19:09:41.680908532+08:00","start-ts":449348837476925447,"target-ts":0,"admin-job-type":1,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":32},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"stopped","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348839626244098} /tidb/cdc/default/default/changefeed/info/changefeed-test3 {"upstream-id":7362131218917024121,"namespace":"default","changefeed-id":"changefeed-test3","sink-uri":"blackhole://","create-time":"2024-04-26T19:09:41.782530652+08:00","start-ts":449348837503139847,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348837503139854} /tidb/cdc/default/default/changefeed/status/changefeed-test1 {"checkpoint-ts":449348841487466507,"min-table-barrier-ts":449348841487466508,"admin-job-type":0} /tidb/cdc/default/default/changefeed/status/changefeed-test2 {"checkpoint-ts":449348837647056911,"min-table-barrier-ts":449348839390314506,"admin-job-type":1} /tidb/cdc/default/default/changefeed/status/changefeed-test3 {"checkpoint-ts":449348841487466507,"min-table-barrier-ts":449348841487466508,"admin-job-type":0} /tidb/cdc/default/default/task/position/3bd47665-7939-4586-9e52-fb9fe566e278/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/3bd47665-7939-4586-9e52-fb9fe566e278/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8611bc78-b9e0-4b0b-836f-ef0915e74f1f/changefeed-test1 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8611bc78-b9e0-4b0b-836f-ef0915e74f1f/changefeed-test3 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362131218917024121 {"id":7362131218917024121,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"address": "127.0.0.1:8300"' VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6863ecc00002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:1135, start at 2024-04-26 19:09:55.889914172 +0800 CST m=+10.267166288 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:55.910 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:55.888 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:55.888 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6863ecc00002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:1135, start at 2024-04-26 19:09:55.889914172 +0800 CST m=+10.267166288 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:55.910 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:55.888 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:55.888 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6863d860000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:1165, start at 2024-04-26 19:09:54.610647505 +0800 CST m=+8.858579112 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:11:54.626 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:09:54.633 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-18:59:54.633 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/row_format/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/row_format/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/row_format/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/row_format/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... "address": "127.0.0.1:8300", run task successfully pass test: list changefeed pass test: get changefeed pass test: resume changefeed + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2376.out cli tso query --pd=http://127.0.0.1:2379 pass test: pause changefeed pass test: rebalance table Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release pass test: move table + set +x + tso='449348842484924417 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348842484924417 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:10:02 CST 2024] <<<<<< START cdc server in row_format case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.24002402.out server --log-file /tmp/tidb_cdc_test/row_format/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/row_format/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) pass test: get processors pass test: list processors pass test: set log level pass test: remove changefeed pass test: resign owner ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:10:05 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c3f0dece-1288-45f0-bf2b-e1568009ed39 {"id":"c3f0dece-1288-45f0-bf2b-e1568009ed39","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a18bca3fc c3f0dece-1288-45f0-bf2b-e1568009ed39 /tidb/cdc/default/default/upstream/7362131352165934024 {"id":7362131352165934024,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c3f0dece-1288-45f0-bf2b-e1568009ed39 {"id":"c3f0dece-1288-45f0-bf2b-e1568009ed39","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a18bca3fc c3f0dece-1288-45f0-bf2b-e1568009ed39 /tidb/cdc/default/default/upstream/7362131352165934024 {"id":7362131352165934024,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c3f0dece-1288-45f0-bf2b-e1568009ed39 {"id":"c3f0dece-1288-45f0-bf2b-e1568009ed39","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a18bca3fc c3f0dece-1288-45f0-bf2b-e1568009ed39 /tidb/cdc/default/default/upstream/7362131352165934024 {"id":7362131352165934024,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.row_format.cli.2438.out cli changefeed create --start-ts=449348842484924417 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 12ff21ef-c776-4087-885b-bdc7db3b8629 Info: {"upstream_id":7362131352165934024,"namespace":"default","id":"12ff21ef-c776-4087-885b-bdc7db3b8629","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:10:06.21342444+08:00","start_ts":449348842484924417,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348842484924417,"checkpoint_ts":449348842484924417,"checkpoint_time":"2024-04-26 19:10:00.739"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6864adec0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:3800, start at 2024-04-26 19:10:08.270503027 +0800 CST m=+5.342799605 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:08.278 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:08.251 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:08.251 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6864adec0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:3800, start at 2024-04-26 19:10:08.270503027 +0800 CST m=+5.342799605 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:08.278 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:08.251 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:08.251 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6864af740005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:3881, start at 2024-04-26 19:10:08.355232317 +0800 CST m=+5.376660181 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:12:08.363 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:10:08.349 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:00:08.349 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/capture_suicide_while_balance_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 26 19:10:13 CST 2024] <<<<<< START cdc server in capture_suicide_while_balance_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_suicide_while_balance_table.51325134.out server --log-file /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:10:16 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7072f0d2-1e48-434f-b742-e51ed4f3d808 {"id":"7072f0d2-1e48-434f-b742-e51ed4f3d808","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900aef3 7072f0d2-1e48-434f-b742-e51ed4f3d808 /tidb/cdc/default/default/upstream/7362131422170276218 {"id":7362131422170276218,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7072f0d2-1e48-434f-b742-e51ed4f3d808 {"id":"7072f0d2-1e48-434f-b742-e51ed4f3d808","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900aef3 7072f0d2-1e48-434f-b742-e51ed4f3d808 /tidb/cdc/default/default/upstream/7362131422170276218 {"id":7362131422170276218,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7072f0d2-1e48-434f-b742-e51ed4f3d808 {"id":"7072f0d2-1e48-434f-b742-e51ed4f3d808","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900aef3 7072f0d2-1e48-434f-b742-e51ed4f3d808 /tidb/cdc/default/default/upstream/7362131422170276218 {"id":7362131422170276218,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:10:16 CST 2024] <<<<<< START cdc server in capture_suicide_while_balance_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=1*return(true)' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_suicide_while_balance_table.51835185.out server --log-file /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_suicide_while_balance_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:10:19 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49c91172-3940-4785-a605-8b25c7af3eed {"id":"49c91172-3940-4785-a605-8b25c7af3eed","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/7072f0d2-1e48-434f-b742-e51ed4f3d808 {"id":"7072f0d2-1e48-434f-b742-e51ed4f3d808","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900aef3 7072f0d2-1e48-434f-b742-e51ed4f3d808 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900af1a 49c91172-3940-4785-a605-8b25c7af3eed /tidb/cdc/default/default/upstream/7362131422170276218 {"id":7362131422170276218,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49c91172-3940-4785-a605-8b25c7af3eed {"id":"49c91172-3940-4785-a605-8b25c7af3eed","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/7072f0d2-1e48-434f-b742-e51ed4f3d808 {"id":"7072f0d2-1e48-434f-b742-e51ed4f3d808","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900aef3 7072f0d2-1e48-434f-b742-e51ed4f3d808 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900af1a 49c91172-3940-4785-a605-8b25c7af3eed /tidb/cdc/default/default/upstream/7362131422170276218 {"id":7362131422170276218,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/49c91172-3940-4785-a605-8b25c7af3eed {"id":"49c91172-3940-4785-a605-8b25c7af3eed","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/7072f0d2-1e48-434f-b742-e51ed4f3d808 {"id":"7072f0d2-1e48-434f-b742-e51ed4f3d808","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900aef3 7072f0d2-1e48-434f-b742-e51ed4f3d808 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1900af1a 49c91172-3940-4785-a605-8b25c7af3eed /tidb/cdc/default/default/upstream/7362131422170276218 {"id":7362131422170276218,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table capture_suicide_while_balance_table.t1 not exists for 1-th check, retry later table capture_suicide_while_balance_table.t1 exists table capture_suicide_while_balance_table.t2 not exists for 1-th check, retry later table capture_suicide_while_balance_table.t2 exists table capture_suicide_while_balance_table.t3 not exists for 1-th check, retry later table row_format.finish_mark not exists for 1-th check, retry later table row_format.finish_mark not exists for 2-th check, retry later table capture_suicide_while_balance_table.t3 not exists for 2-th check, retry later table row_format.finish_mark not exists for 3-th check, retry later table capture_suicide_while_balance_table.t3 exists table capture_suicide_while_balance_table.t4 not exists for 1-th check, retry later table row_format.finish_mark not exists for 4-th check, retry later table capture_suicide_while_balance_table.t4 exists table row_format.finish_mark not exists for 5-th check, retry later table row_format.finish_mark not exists for 6-th check, retry later % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 140 100 35 100 105 785 2355 --:--:-- --:--:-- --:--:-- 2386 { "status": true, "message": "" }lease 22318f1a1900af1a revoked check diff successfully table row_format.finish_mark not exists for 7-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:10:36 CST 2024] <<<<<< run test case capture_suicide_while_balance_table success! >>>>>> pass test: get tso wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... table row_format.finish_mark not exists for 8-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:10:37 CST 2024] <<<<<< run test case http_api success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 37.16 secs (90551574 bytes/sec) [Pipeline] { [Pipeline] cache table row_format.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:10:41 CST 2024] <<<<<< run test case row_format success! >>>>>> Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 5.85 secs (574976159 bytes/sec) [Pipeline] { [Pipeline] cache =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_api_tls/run.sh using Sink-Type: mysql... <<================= WARNING: Running pip install with root privileges is generally not a good idea. Try `__main__.py install --user` instead. Requirement already up-to-date: requests==2.26.0 in /usr/local/lib/python3.6/site-packages Requirement already up-to-date: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: certifi>=2017.4.17 in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: charset-normalizer~=2.0.0; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) Requirement already up-to-date: idna<4,>=2.5; python_version >= "3" in /usr/local/lib/python3.6/site-packages (from requests==2.26.0) The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/syncpoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/syncpoint Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/drop_many_tables/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/drop_many_tables Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6867f0880013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:4039, start at 2024-04-26 19:11:01.706898657 +0800 CST m=+5.345235098 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:01.714 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:01.716 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:01.716 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6867f0880013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:4039, start at 2024-04-26 19:11:01.706898657 +0800 CST m=+5.345235098 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:01.714 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:01.716 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:01.716 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6867f2b40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:4106, start at 2024-04-26 19:11:01.806329633 +0800 CST m=+5.374375408 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:01.813 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:01.805 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:01.805 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_api_tls/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_api_tls/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_api_tls/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) The 1 times to try to start tls tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/http_api_tls Starting TLS PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 18.52 secs (181673947 bytes/sec) Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686840cc0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:6501, start at 2024-04-26 19:11:06.82351575 +0800 CST m=+5.226679036 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:06.831 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:06.803 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:06.803 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686840cc0010 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:6501, start at 2024-04-26 19:11:06.82351575 +0800 CST m=+5.226679036 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:06.831 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:06.803 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:06.803 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686840bc0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:6582, start at 2024-04-26 19:11:06.824210481 +0800 CST m=+5.173693668 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:06.832 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:06.799 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:06.799 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/syncpoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/syncpoint/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/syncpoint/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting TLS TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.cli.7868.out cli tso query --pd=http://127.0.0.1:2379 Starting TLS TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying TLS TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449348861415391233 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348861415391233 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:11:14 CST 2024] <<<<<< START cdc server in syncpoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/DDLSinkInitializeSlowly=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.79067908.out server --log-file /tmp/tidb_cdc_test/syncpoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/syncpoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 5.68 secs (592349462 bytes/sec) [Pipeline] { [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6868bec8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:3449, start at 2024-04-26 19:11:14.906056986 +0800 CST m=+7.930633730 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:14.928 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:14.916 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:14.916 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6868bec8000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:3449, start at 2024-04-26 19:11:14.906056986 +0800 CST m=+7.930633730 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:14.928 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:14.916 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:14.916 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:11:17 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6e050269-4778-4b49-a367-461d6fe5111f {"id":"6e050269-4778-4b49-a367-461d6fe5111f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19dc3703 6e050269-4778-4b49-a367-461d6fe5111f /tidb/cdc/default/default/upstream/7362131673417996020 {"id":7362131673417996020,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6e050269-4778-4b49-a367-461d6fe5111f {"id":"6e050269-4778-4b49-a367-461d6fe5111f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19dc3703 6e050269-4778-4b49-a367-461d6fe5111f /tidb/cdc/default/default/upstream/7362131673417996020 {"id":7362131673417996020,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/6e050269-4778-4b49-a367-461d6fe5111f {"id":"6e050269-4778-4b49-a367-461d6fe5111f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19dc3703 6e050269-4778-4b49-a367-461d6fe5111f /tidb/cdc/default/default/upstream/7362131673417996020 {"id":7362131673417996020,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.syncpoint.cli.7965.out cli changefeed create --start-ts=449348861415391233 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/syncpoint/conf/changefeed.toml Create changefeed successfully! ID: 9f47e3f8-c62f-4cb0-a3e4-7eb636b92c7a Info: {"upstream_id":7362131673417996020,"namespace":"default","id":"9f47e3f8-c62f-4cb0-a3e4-7eb636b92c7a","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:11:17.876195251+08:00","start_ts":449348861415391233,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":true,"bdr_mode":false,"sync_point_interval":30000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348861415391233,"checkpoint_ts":449348861415391233,"checkpoint_time":"2024-04-26 19:11:12.953"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6868f0a80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:5609, start at 2024-04-26 19:11:18.101000973 +0800 CST m=+5.456398587 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:18.109 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:18.108 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:18.108 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6868c0180006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:3482, start at 2024-04-26 19:11:14.967597849 +0800 CST m=+7.891932701 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:13:14.988 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:11:14.999 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:01:14.999 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [Fri Apr 26 19:11:18 CST 2024] <<<<<< START cdc server in http_api_tls case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls.56525654.out server --log-file /tmp/tidb_cdc_test/http_api_tls/cdc_http_api_tls_tls1.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls/cdc_data_http_api_tls_tls1 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8300 --pd https://127.0.0.1:2579 ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/drop_many_tables/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/drop_many_tables/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x ***************** properties ***************** "workload"="core" "operationcount"="0" "mysql.db"="testSync" "threadcount"="2" "mysql.user"="root" "mysql.host"="127.0.0.1" "dotransactions"="false" "insertproportion"="0" "requestdistribution"="uniform" "recordcount"="10" "mysql.port"="4000" "updateproportion"="0" "readallfields"="true" "readproportion"="0" "scanproportion"="0" ********************************************** Run finished, takes 10.836078ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1921.4, Avg(us): 2078, Min(us): 1080, Max(us): 5537, 95th(us): 6000, 99th(us): 6000 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.4695.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none * NSS: client certificate from file * subject: CN=client * start date: Feb 18 07:48:00 2020 GMT * expire date: Jan 25 07:48:00 2120 GMT * common name: client * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * common name: tidb-server * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:11:21 GMT < Content-Length: 1067 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fce27e81-ebe0-41f5-b09a-5dc6812bbcef {"id":"fce27e81-ebe0-41f5-b09a-5dc6812bbcef","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c0f fce27e81-ebe0-41f5-b09a-5dc6812bbcef /tidb/cdc/default/default/upstream/7362131732712171478 {"id":7362131732712171478,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fce27e81-ebe0-41f5-b09a-5dc6812bbcef {"id":"fce27e81-ebe0-41f5-b09a-5dc6812bbcef","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c0f fce27e81-ebe0-41f5-b09a-5dc6812bbcef /tidb/cdc/default/default/upstream/7362131732712171478 {"id":7362131732712171478,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fce27e81-ebe0-41f5-b09a-5dc6812bbcef {"id":"fce27e81-ebe0-41f5-b09a-5dc6812bbcef","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c0f fce27e81-ebe0-41f5-b09a-5dc6812bbcef /tidb/cdc/default/default/upstream/7362131732712171478 {"id":7362131732712171478,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x + set +x + tso='449348863725666308 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348863725666308 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:11:23 CST 2024] <<<<<< START cdc server in drop_many_tables case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.47104712.out server --log-file /tmp/tidb_cdc_test/drop_many_tables/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/drop_many_tables/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 26 19:11:23 CST 2024] <<<<<< START cdc server in http_api_tls case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates ']' + curl_status_cmd='curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_api_tls.57005702.out server --log-file /tmp/tidb_cdc_test/http_api_tls/cdc_http_api_tls_tls2.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_api_tls/cdc_data_http_api_tls_tls2 --cluster-id default --config /tmp/tidb_cdc_test/http_api_tls/server.toml --ca /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem --cert-allowed-cn client --addr 127.0.0.1:8301 --pd https://127.0.0.1:2579 + (( i = 0 )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:11:26 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff314ace-4746-4ade-bc66-6f37d08670b3 {"id":"ff314ace-4746-4ade-bc66-6f37d08670b3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19faa8fb ff314ace-4746-4ade-bc66-6f37d08670b3 /tidb/cdc/default/default/upstream/7362131706129604003 {"id":7362131706129604003,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff314ace-4746-4ade-bc66-6f37d08670b3 {"id":"ff314ace-4746-4ade-bc66-6f37d08670b3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19faa8fb ff314ace-4746-4ade-bc66-6f37d08670b3 /tidb/cdc/default/default/upstream/7362131706129604003 {"id":7362131706129604003,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ff314ace-4746-4ade-bc66-6f37d08670b3 {"id":"ff314ace-4746-4ade-bc66-6f37d08670b3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a19faa8fb ff314ace-4746-4ade-bc66-6f37d08670b3 /tidb/cdc/default/default/upstream/7362131706129604003 {"id":7362131706129604003,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.drop_many_tables.cli.4747.out cli changefeed create --start-ts=449348863725666308 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ + (( i++ )) + (( i <= 50 )) ++ curl --cacert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem -vsL --max-time 20 https://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem CApath: none * NSS: client certificate from file * subject: CN=client * start date: Feb 18 07:48:00 2020 GMT * expire date: Jan 25 07:48:00 2120 GMT * common name: client * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=tidb-server * start date: Feb 18 09:11:00 2020 GMT * expire date: Jan 25 09:11:00 2120 GMT * common name: tidb-server * issuer: CN=My own CA,O=PingCAP,L=Beijing,ST=Beijing,C=CN > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:11:27 GMT < Content-Length: 1321 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eea496e9-a399-4a2c-98c3-85d6ba2b04c8 {"id":"eea496e9-a399-4a2c-98c3-85d6ba2b04c8","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/fce27e81-ebe0-41f5-b09a-5dc6812bbcef {"id":"fce27e81-ebe0-41f5-b09a-5dc6812bbcef","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c0f fce27e81-ebe0-41f5-b09a-5dc6812bbcef /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c3c eea496e9-a399-4a2c-98c3-85d6ba2b04c8 /tidb/cdc/default/default/upstream/7362131732712171478 {"id":7362131732712171478,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eea496e9-a399-4a2c-98c3-85d6ba2b04c8 {"id":"eea496e9-a399-4a2c-98c3-85d6ba2b04c8","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/fce27e81-ebe0-41f5-b09a-5dc6812bbcef {"id":"fce27e81-ebe0-41f5-b09a-5dc6812bbcef","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c0f fce27e81-ebe0-41f5-b09a-5dc6812bbcef /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c3c eea496e9-a399-4a2c-98c3-85d6ba2b04c8 /tidb/cdc/default/default/upstream/7362131732712171478 {"id":7362131732712171478,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/eea496e9-a399-4a2c-98c3-85d6ba2b04c8 {"id":"eea496e9-a399-4a2c-98c3-85d6ba2b04c8","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/fce27e81-ebe0-41f5-b09a-5dc6812bbcef {"id":"fce27e81-ebe0-41f5-b09a-5dc6812bbcef","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c0f fce27e81-ebe0-41f5-b09a-5dc6812bbcef /tidb/cdc/default/__cdc_meta__/owner/1f5d8f1a1a105c3c eea496e9-a399-4a2c-98c3-85d6ba2b04c8 /tidb/cdc/default/default/upstream/7362131732712171478 {"id":7362131732712171478,"pd-endpoints":"https://127.0.0.1:2579,https://127.0.0.1:2579","key-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server-key.pem","cert-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/server.pem","ca-path":"/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem","cert-allowed-cn":["client","tidb-server"]}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 0b1b22c2-d3e9-4ab4-864a-cc540205788a Info: {"upstream_id":7362131706129604003,"namespace":"default","id":"0b1b22c2-d3e9-4ab4-864a-cc540205788a","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:11:27.296217286+08:00","start_ts":449348863725666308,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348863725666308,"checkpoint_ts":449348863725666308,"checkpoint_time":"2024-04-26 19:11:21.766"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... + set +x pass test: check health pass test: get status pass test: create changefeed check_changefeed_state https://127.0.0.1:2579 changefeed-test1 normal null /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test1 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test1 -s + info='{ "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 449348865794506759, "checkpoint_time": "2024-04-26 19:11:29.658", "error": null }' + echo '{ "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 449348865794506759, "checkpoint_time": "2024-04-26 19:11:29.658", "error": null }' { "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test1", "state": "normal", "checkpoint_tso": 449348865794506759, "checkpoint_time": "2024-04-26 19:11:29.658", "error": null } ++ echo '{' '"upstream_id":' 7362131732712171478, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 449348865794506759, '"checkpoint_time":' '"2024-04-26' '19:11:29.658",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362131732712171478, '"namespace":' '"default",' '"id":' '"changefeed-test1",' '"state":' '"normal",' '"checkpoint_tso":' 449348865794506759, '"checkpoint_time":' '"2024-04-26' '19:11:29.658",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test2 normal null /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test2 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test2 -s + info='{ "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 449348865860042758, "checkpoint_time": "2024-04-26 19:11:29.908", "error": null }' + echo '{ "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 449348865860042758, "checkpoint_time": "2024-04-26 19:11:29.908", "error": null }' { "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test2", "state": "normal", "checkpoint_tso": 449348865860042758, "checkpoint_time": "2024-04-26 19:11:29.908", "error": null } ++ echo '{' '"upstream_id":' 7362131732712171478, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 449348865860042758, '"checkpoint_time":' '"2024-04-26' '19:11:29.908",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362131732712171478, '"namespace":' '"default",' '"id":' '"changefeed-test2",' '"state":' '"normal",' '"checkpoint_tso":' 449348865860042758, '"checkpoint_time":' '"2024-04-26' '19:11:29.908",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test3 normal null /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test3 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test3 -s + info='{ "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 449348865899364372, "checkpoint_time": "2024-04-26 19:11:30.058", "error": null }' + echo '{ "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 449348865899364372, "checkpoint_time": "2024-04-26 19:11:30.058", "error": null }' { "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test3", "state": "normal", "checkpoint_tso": 449348865899364372, "checkpoint_time": "2024-04-26 19:11:30.058", "error": null } ++ echo '{' '"upstream_id":' 7362131732712171478, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 449348865899364372, '"checkpoint_time":' '"2024-04-26' '19:11:30.058",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362131732712171478, '"namespace":' '"default",' '"id":' '"changefeed-test3",' '"state":' '"normal",' '"checkpoint_tso":' 449348865899364372, '"checkpoint_time":' '"2024-04-26' '19:11:30.058",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state https://127.0.0.1:2579 changefeed-test4 normal null /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + endpoints=https://127.0.0.1:2579 + changefeed_id=changefeed-test4 + expected_state=normal + error_msg=null + tls_dir=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates + [[ https://127.0.0.1:2579 =~ https ]] ++ cdc cli changefeed query --ca=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/ca.pem --cert=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client.pem --key=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_certificates/client-key.pem --pd=https://127.0.0.1:2579 -c changefeed-test4 -s + info='{ "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 449348865964900359, "checkpoint_time": "2024-04-26 19:11:30.308", "error": null }' + echo '{ "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 449348865964900359, "checkpoint_time": "2024-04-26 19:11:30.308", "error": null }' { "upstream_id": 7362131732712171478, "namespace": "default", "id": "changefeed-test4", "state": "normal", "checkpoint_tso": 449348865964900359, "checkpoint_time": "2024-04-26 19:11:30.308", "error": null } ++ echo '{' '"upstream_id":' 7362131732712171478, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 449348865964900359, '"checkpoint_time":' '"2024-04-26' '19:11:30.308",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362131732712171478, '"namespace":' '"default",' '"id":' '"changefeed-test4",' '"state":' '"normal",' '"checkpoint_tso":' 449348865964900359, '"checkpoint_time":' '"2024-04-26' '19:11:30.308",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully table drop_tables.c not exists for 1-th check, retry later table drop_tables.c not exists for 2-th check, retry later table test.simple1 exists pass test: list changefeed pass test: get changefeed table drop_tables.c not exists for 3-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 17.62 secs (190996897 bytes/sec) [Pipeline] { [Pipeline] cache pass test: pause changefeed table drop_tables.c not exists for 4-th check, retry later pass test: update changefeed table drop_tables.c exists check diff successfully pass test: resume changefeed wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... pass test: rebalance table cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:11:43 CST 2024] <<<<<< run test case drop_many_tables success! >>>>>> pass test: list processors Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 7.01 secs (480065181 bytes/sec) [Pipeline] { [Pipeline] cache pass test: get processors pass test: move table pass test: set log level pass test: remove changefeed pass test: resign owner pass test: get tso pass test: verify table =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_stop_delay/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 6.99 secs (481130103 bytes/sec) [Pipeline] { [Pipeline] cache pass test: create changefeed v2 pass test: remove changefeed v2 start tidb cluster in /tmp/tidb_cdc_test/processor_stop_delay Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... status code 200 pass test: show metadata status code 200 status code 200 pass test: delete service_gc_safepoint status code 500 pass test: resolve lock wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:12:03 CST 2024] <<<<<< run test case http_api_tls success! >>>>>> Agent ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g is provisioned from template ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-fw3w4-xxnpf --- apiVersion: "v1" kind: "Pod" metadata: annotations: buildUrl: "http://jenkins.apps.svc.cluster.local:8080/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" runUrl: "job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/" labels: jenkins/jenkins-jenkins-agent: "true" jenkins/label-digest: "a0f34cbf6109d0545ad5b6f73c71860789da8ce2" jenkins/label: "ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-fw3w4" name: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g" namespace: "jenkins-tiflow" spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: "kubernetes.io/arch" operator: "In" values: - "amd64" containers: - image: "hub.pingcap.net/jenkins/centos7_golang-1.21:latest" name: "golang" resources: limits: memory: "16Gi" cpu: "6" requests: memory: "12Gi" cpu: "4" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/network-multitool" name: "net-tool" resources: limits: memory: "128Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - image: "hub.pingcap.net/jenkins/python3-requests:latest" name: "report" resources: limits: memory: "256Mi" cpu: "100m" tty: true volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_TUNNEL" value: "jenkins-agent.apps.svc.cluster.local:50000" - name: "JENKINS_AGENT_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g" - name: "JENKINS_NAME" value: "ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.apps.svc.cluster.local:8080/jenkins/" image: "jenkins/inbound-agent:3206.vb_15dcf73f6a_9-2" name: "jnlp" resources: requests: memory: "256Mi" cpu: "100m" volumeMounts: - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false restartPolicy: "Never" securityContext: fsGroup: 1000 volumes: - emptyDir: medium: "" name: "workspace-volume" Running on ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/api_v2/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/api_v2 Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c56cc0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:5696, start at 2024-04-26 19:12:13.753995184 +0800 CST m=+7.544046880 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:13.773 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:13.747 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:13.747 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c56cc0003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:5696, start at 2024-04-26 19:12:13.753995184 +0800 CST m=+7.544046880 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:13.773 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:13.747 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:13.747 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686c5a300003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:5720, start at 2024-04-26 19:12:13.966187525 +0800 CST m=+7.621653440 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:13.989 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:13.964 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:13.964 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/error.log Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_stop_delay/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 26 19:12:21 CST 2024] <<<<<< START cdc server in processor_stop_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorStopDelay=1*sleep(10000)' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_stop_delay.69376939.out server --log-file /tmp/tidb_cdc_test/processor_stop_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_stop_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:24 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aa8e3324-93ae-44f6-8edd-b6d86d6e9986 {"id":"aa8e3324-93ae-44f6-8edd-b6d86d6e9986","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1adff3f8 aa8e3324-93ae-44f6-8edd-b6d86d6e9986 /tidb/cdc/default/default/upstream/7362131957306217127 {"id":7362131957306217127,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aa8e3324-93ae-44f6-8edd-b6d86d6e9986 {"id":"aa8e3324-93ae-44f6-8edd-b6d86d6e9986","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1adff3f8 aa8e3324-93ae-44f6-8edd-b6d86d6e9986 /tidb/cdc/default/default/upstream/7362131957306217127 {"id":7362131957306217127,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/aa8e3324-93ae-44f6-8edd-b6d86d6e9986 {"id":"aa8e3324-93ae-44f6-8edd-b6d86d6e9986","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1adff3f8 aa8e3324-93ae-44f6-8edd-b6d86d6e9986 /tidb/cdc/default/default/upstream/7362131957306217127 {"id":7362131957306217127,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_stop_delay.t not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686cfe680004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:6870, start at 2024-04-26 19:12:24.477912801 +0800 CST m=+5.311565941 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:24.485 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:24.474 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:24.474 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686cfe680004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:6870, start at 2024-04-26 19:12:24.477912801 +0800 CST m=+5.311565941 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:24.485 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:24.474 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:24.474 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686cffe00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:6944, start at 2024-04-26 19:12:24.572247206 +0800 CST m=+5.333377175 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:24.579 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:24.568 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:24.568 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... ***************** properties ***************** "threadcount"="2" "recordcount"="10" "mysql.db"="testSync" "readallfields"="true" "scanproportion"="0" "dotransactions"="false" "updateproportion"="0" "mysql.host"="127.0.0.1" "workload"="core" "operationcount"="0" "insertproportion"="0" "mysql.user"="root" "mysql.port"="4000" "requestdistribution"="uniform" "readproportion"="0" ********************************************** Run finished, takes 4.05969ms INSERT - Takes(s): 0.0, Count: 10, OPS: 3518.1, Avg(us): 714, Min(us): 440, Max(us): 1724, 95th(us): 2000, 99th(us): 2000 Logging trace to /tmp/tidb_cdc_test/api_v2/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/api_v2/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/api_v2/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table processor_stop_delay.t exists check diff failed 1-th time, retry later [Fri Apr 26 19:12:30 CST 2024] <<<<<< START cdc server in api_v2 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.api_v2.83048306.out server --log-file /tmp/tidb_cdc_test/api_v2/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/api_v2/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:33 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c14ec8-9682-4260-95d3-d7d543ae14f0 {"id":"43c14ec8-9682-4260-95d3-d7d543ae14f0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b1cd0ff 43c14ec8-9682-4260-95d3-d7d543ae14f0 /tidb/cdc/default/default/upstream/7362132018901525849 {"id":7362132018901525849,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c14ec8-9682-4260-95d3-d7d543ae14f0 {"id":"43c14ec8-9682-4260-95d3-d7d543ae14f0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b1cd0ff 43c14ec8-9682-4260-95d3-d7d543ae14f0 /tidb/cdc/default/default/upstream/7362132018901525849 {"id":7362132018901525849,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/43c14ec8-9682-4260-95d3-d7d543ae14f0 {"id":"43c14ec8-9682-4260-95d3-d7d543ae14f0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b1cd0ff 43c14ec8-9682-4260-95d3-d7d543ae14f0 /tidb/cdc/default/default/upstream/7362132018901525849 {"id":7362132018901525849,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: cf-blackhole Info: {"upstream_id":7362132018901525849,"namespace":"default","id":"cf-blackhole","sink_uri":"blackhole:","create_time":"2024-04-26T19:12:33.456430115+08:00","start_ts":449348882484166660,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348882484166660,"checkpoint_ts":449348882484166660,"checkpoint_time":"2024-04-26 19:12:33.324"} + endpoints=http://127.0.0.1:2379 + changefeed_id=cf-blackhole + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c cf-blackhole -s + info='{ "upstream_id": 7362132018901525849, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 449348882484166660, "checkpoint_time": "2024-04-26 19:12:33.324", "error": null }' + echo '{ "upstream_id": 7362132018901525849, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 449348882484166660, "checkpoint_time": "2024-04-26 19:12:33.324", "error": null }' { "upstream_id": 7362132018901525849, "namespace": "default", "id": "cf-blackhole", "state": "normal", "checkpoint_tso": 449348882484166660, "checkpoint_time": "2024-04-26 19:12:33.324", "error": null } ++ echo '{' '"upstream_id":' 7362132018901525849, '"namespace":' '"default",' '"id":' '"cf-blackhole",' '"state":' '"normal",' '"checkpoint_tso":' 449348882484166660, '"checkpoint_time":' '"2024-04-26' '19:12:33.324",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132018901525849, '"namespace":' '"default",' '"id":' '"cf-blackhole",' '"state":' '"normal",' '"checkpoint_tso":' 449348882484166660, '"checkpoint_time":' '"2024-04-26' '19:12:33.324",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + GO111MODULE=on + go run main.go model.go request.go cases.go go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading go.uber.org/zap v1.27.0 go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/gin-gonic/gin v1.8.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/IBM/sarama v1.41.2 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading golang.org/x/sync v0.7.0 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading github.com/uber-go/atomic v1.4.0 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/goccy/go-json v0.10.2 go: downloading github.com/stretchr/testify v1.9.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading cloud.google.com/go v0.112.0 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/klauspost/compress v1.17.7 check diff failed 2-th time, retry later go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading blainsmith.com/go/seahash v1.2.1 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading golang.org/x/crypto v0.22.0 Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 33.93 secs (99177989 bytes/sec) [Pipeline] { [Pipeline] sh go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading github.com/gin-contrib/sse v0.1.0 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading golang.org/x/net v0.24.0 go: downloading github.com/gavv/monotime v0.0.0-20190418164738-30dba4353424 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading google.golang.org/protobuf v1.33.0 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading golang.org/x/sys v0.19.0 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 [Pipeline] sh go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20230619063055-fd67d94318fd go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading github.com/stretchr/objx v0.5.2 go: downloading cloud.google.com/go/compute/metadata v0.2.3 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G04 [Pipeline] sh go: downloading cloud.google.com/go/iam v1.1.6 go: downloading cloud.google.com/go/compute v1.24.0 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.162.0 Run cases: foreign_key ddl_puller_lag ddl_only_block_related_table changefeed_auto_stop PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=6052cab6-cc0b-4fb6-823b-13a9f9e511a9 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G04 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-8jzdv GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-8jzdv ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/foreign_key/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G05 Run cases: charset_gbk ddl_manager multi_source PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=5dbc1f8c-1527-47b0-bc96-7981c76e59e8 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G05 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-4qcfc GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-4qcfc GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/charset_gbk/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G07 Run cases: kv_client_stream_reconnect cdc split_region PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=54e54500-61be-4ef3-a8ad-bee4e643711d BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G07 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-rv597 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-rv597 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kv_client_stream_reconnect/run.sh using Sink-Type: mysql... <<================= + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G06 Run cases: sink_retry changefeed_error ddl_sequence resourcecontrol PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=b562b263-1d44-4553-b307-16fbffad0de0 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G06 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-nnw0d GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-nnw0d GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/sink_retry/run.sh using Sink-Type: mysql... <<================= [Pipeline] sh The 1 times to try to start tidb cluster... The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G09 [Pipeline] sh Run cases: gc_safepoint changefeed_pause_resume cli savepoint synced_status PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=2c733287-1a95-4c26-842d-a6bdaf3fd0f6 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G09 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-fz0c2 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-fz0c2 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/gc_safepoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading github.com/pierrec/lz4 v2.6.1+incompatible + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G08 Run cases: processor_err_chan changefeed_reconstruct multi_capture PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=64548d37-2689-4540-815f-c2e0df8042c7 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G08 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-7rhcc GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-7rhcc ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_err_chan/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh check diff failed 3-th time, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G10 Run cases: default_value simple cdc_server_tips event_filter PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=473ce64e-e90c-498b-aeb5-de2b74c13094 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G10 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-1p0j0 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-1p0j0 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/default_value/run.sh using Sink-Type: mysql... <<================= [Pipeline] sh go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading go.etcd.io/bbolt v1.3.8 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading golang.org/x/text v0.14.0 go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/hashicorp/golang-lru v0.5.1 go: downloading github.com/go-playground/validator/v10 v10.11.1 go: downloading github.com/pelletier/go-toml/v2 v2.0.5 go: downloading github.com/ugorji/go/codec v1.2.7 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/imdario/mergo v0.3.16 go: downloading github.com/segmentio/kafka-go v0.4.41-0.20230526171612-f057b1d369cd go: downloading github.com/DATA-DOG/go-sqlmock v1.5.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/shopspring/decimal v1.3.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G02 Run cases: consistent_replicate_ddl consistent_replicate_gbk consistent_replicate_nfs consistent_replicate_storage_file consistent_replicate_storage_file_large_value consistent_replicate_storage_s3 consistent_partition_table kafka_big_txn_v2 kafka_big_messages_v2 multi_tables_ddl_v2 multi_topics_v2 storage_cleanup csv_storage_basic csv_storage_multi_tables_ddl csv_storage_partition_table PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=9254e849-ccc4-486d-8a76-cc0dcfea6a89 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G02 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-4gkcs GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-4gkcs GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_ddl/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/otiai10/copy v1.2.0 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/gorilla/websocket v1.5.1 + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G11 Run cases: resolve_lock move_table autorandom generate_column PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=fb04d3e0-c65f-4f0f-a9c3-4da6018b6c2f BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G11 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-hz3f1 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-hz3f1 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/resolve_lock/run.sh using Sink-Type: mysql... <<================= go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0 go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/go-playground/universal-translator v0.18.0 go: downloading github.com/leodido/go-urn v1.2.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/xdg-go/scram v1.1.2 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/aws/aws-sdk-go-v2/config v1.18.30 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.29 go: downloading github.com/aws/aws-sdk-go-v2/service/glue v1.58.1 go: downloading github.com/jarcoal/httpmock v1.2.0 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/blacktear23/go-proxyprotocol v1.0.6 go: downloading github.com/pingcap/fn v1.0.0 go: downloading github.com/tiancaiamao/appdash v0.0.0-20181126055449-889f96f722a2 go: downloading github.com/sourcegraph/appdash-data v0.0.0-20151005221446-73f23eafcf67 go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/go-playground/locales v0.14.0 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c go: downloading github.com/mtibben/percent v0.2.1 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/xdg-go/pbkdf2 v1.0.0 go: downloading github.com/xdg-go/stringprep v1.0.4 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.37 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.12.14 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.14 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.20.1 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.36 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.30 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/rivo/uniseg v0.4.6 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.30 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 check diff failed 4-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/sink_retry Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... go: downloading k8s.io/apimachinery v0.28.6 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b start tidb cluster in /tmp/tidb_cdc_test/foreign_key Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/processor_err_chan Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... go: downloading github.com/ardielle/ardielle-go v1.5.2 The 1 times to try to start tidb cluster... go: downloading github.com/jmespath/go-jmespath v0.4.0 start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_ddl Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading github.com/felixge/httpsnoop v1.0.4 check diff failed 5-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/gc_safepoint Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { [Pipeline] withEnv [Pipeline] { check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Verifying downstream PD is started... wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/default_value Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:12:45 CST 2024] <<<<<< run test case processor_stop_delay success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { [Pipeline] container [Pipeline] { Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/kv_client_stream_reconnect Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) start tidb cluster in /tmp/tidb_cdc_test/charset_gbk Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [Pipeline] stage [Pipeline] { (Test) [Pipeline] stage [Pipeline] { (Test) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] { ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] checkout [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { The recommended git tool is: git [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] withCredentials No credentials specified Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@7b760db3; decorates RemoteLauncher[hudson.remoting.Channel@31eec8a6:JNLP4-connect connection from 10.233.100.201/10.233.100.201:51580] will be ignored (a typical symptom is the Git executable not being run inside a designated container) Cloning the remote Git repository Using shallow clone with depth 1 Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN Cloning repository https://github.com/PingCAP-QE/ci.git > git init /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test # timeout=10 Fetching upstream changes from https://github.com/PingCAP-QE/ci.git > git --version # timeout=10 > git --version # 'git version 2.39.2' > git fetch --tags --force --progress --depth=1 -- https://github.com/PingCAP-QE/ci.git +refs/heads/*:refs/remotes/origin/* # timeout=5 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { [Pipeline] { ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] cache ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686e9e140018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:1178, start at 2024-04-26 19:12:51.123312825 +0800 CST m=+5.388970195 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:51.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:51.127 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:51.127 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686e9e140018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:1178, start at 2024-04-26 19:12:51.123312825 +0800 CST m=+5.388970195 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:51.130 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:51.127 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:51.127 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686e9fa0000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:1253, start at 2024-04-26 19:12:51.19272721 +0800 CST m=+5.406295619 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:51.199 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:51.176 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:51.176 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_err_chan/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_err_chan/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } > git config remote.origin.url https://github.com/PingCAP-QE/ci.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 start tidb cluster in /tmp/tidb_cdc_test/resolve_lock Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Avoid second fetch Checking out Revision cb9bc9e8822a01a5d59a2f670fb429e588065145 (origin/main) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Commit message: "feat(tidb): test flashbacktest package (#2942)" VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eac300014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:1231, start at 2024-04-26 19:12:52.022695619 +0800 CST m=+5.252517488 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.029 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.030 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.030 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eac300014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:1231, start at 2024-04-26 19:12:52.022695619 +0800 CST m=+5.252517488 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.029 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.030 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.030 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eae780011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:1311, start at 2024-04-26 19:12:52.144185442 +0800 CST m=+5.325774109 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.153 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.126 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.126 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_ddl/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) > git rev-parse origin/main^{commit} # timeout=10 > git config core.sparsecheckout # timeout=10 > git checkout -f cb9bc9e8822a01a5d59a2f670fb429e588065145 # timeout=10 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb0000017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:1171, start at 2024-04-26 19:12:52.250814905 +0800 CST m=+5.244381841 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.258 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.224 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.224 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb0000017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:1171, start at 2024-04-26 19:12:52.250814905 +0800 CST m=+5.244381841 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.258 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.224 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.224 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb1880004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:1253, start at 2024-04-26 19:12:52.325298848 +0800 CST m=+5.218842600 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.333 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.322 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.322 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb414000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:1135, start at 2024-04-26 19:12:52.512951318 +0800 CST m=+7.286156684 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.531 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.535 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.535 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb414000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:1135, start at 2024-04-26 19:12:52.512951318 +0800 CST m=+7.286156684 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.531 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.535 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.535 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb448000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:1170, start at 2024-04-26 19:12:52.528002421 +0800 CST m=+7.198571006 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.542 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.544 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.544 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/gc_safepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/gc_safepoint/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/foreign_key/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/foreign_key/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/foreign_key/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb600000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:1169, start at 2024-04-26 19:12:52.618804248 +0800 CST m=+8.396212521 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.628 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.608 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.608 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb600000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:1169, start at 2024-04-26 19:12:52.618804248 +0800 CST m=+8.396212521 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.628 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.608 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.608 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686eb9400014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:1247, start at 2024-04-26 19:12:52.865037786 +0800 CST m=+8.572707233 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:52.871 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:52.866 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:52.866 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_retry/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_retry/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_retry/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Fri Apr 26 19:12:55 CST 2024] <<<<<< START cdc server in consistent_replicate_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_ddl.25772579.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_ddl/cdcconsistent_replicate_ddl.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_ddl/cdc_dataconsistent_replicate_ddl.server1 --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2365.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:12:55 CST 2024] <<<<<< START cdc server in gc_safepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectGcSafepointUpdateInterval=return(500)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.gc_safepoint.25672569.out server --log-file /tmp/tidb_cdc_test/gc_safepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/gc_safepoint/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686edd100018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:1291, start at 2024-04-26 19:12:55.156164491 +0800 CST m=+5.219793051 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:55.167 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:55.159 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:55.159 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686edd100018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:1291, start at 2024-04-26 19:12:55.156164491 +0800 CST m=+5.219793051 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:55.167 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:55.159 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:55.159 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ede840008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:1357, start at 2024-04-26 19:12:55.210912526 +0800 CST m=+5.218096523 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:55.219 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:55.201 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:55.201 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/default_value/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/default_value/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/default_value/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/default_value/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/default_value/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2565.out cli tso query --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/partition_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.default_value.cli.2579.out cli tso query --pd=http://127.0.0.1:2379 < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:58 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/924391be-6287-4364-bd37-ad225bf36bfb {"id":"924391be-6287-4364-bd37-ad225bf36bfb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ee1f2 924391be-6287-4364-bd37-ad225bf36bfb /tidb/cdc/default/default/upstream/7362132132842926423 {"id":7362132132842926423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/924391be-6287-4364-bd37-ad225bf36bfb {"id":"924391be-6287-4364-bd37-ad225bf36bfb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ee1f2 924391be-6287-4364-bd37-ad225bf36bfb /tidb/cdc/default/default/upstream/7362132132842926423 {"id":7362132132842926423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/924391be-6287-4364-bd37-ad225bf36bfb {"id":"924391be-6287-4364-bd37-ad225bf36bfb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ee1f2 924391be-6287-4364-bd37-ad225bf36bfb /tidb/cdc/default/default/upstream/7362132132842926423 {"id":7362132132842926423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + set +x + tso='449348888679677953 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348888679677953 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + set +x + tso='449348888516624388 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348888516624388 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:12:58 CST 2024] <<<<<< START cdc server in foreign_key case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.23852387.out server --log-file /tmp/tidb_cdc_test/foreign_key/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/foreign_key/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ***************** properties ***************** "workload"="core" "insertproportion"="0" "requestdistribution"="uniform" "mysql.db"="sink_retry" "threadcount"="2" "readproportion"="0" "scanproportion"="0" "mysql.port"="4000" "readallfields"="true" "recordcount"="10" "mysql.host"="127.0.0.1" "updateproportion"="0" "dotransactions"="false" "operationcount"="0" "mysql.user"="root" ********************************************** Run finished, takes 10.020162ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1848.9, Avg(us): 1906, Min(us): 1036, Max(us): 4476, 95th(us): 5000, 99th(us): 5000 [Fri Apr 26 19:12:58 CST 2024] <<<<<< START cdc server in sink_retry case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkTxnRandomError=25%return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.26222624.out server --log-file /tmp/tidb_cdc_test/sink_retry/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_retry/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ef6640006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:1226, start at 2024-04-26 19:12:56.737941139 +0800 CST m=+5.567492253 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:56.746 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:56.729 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:56.729 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ef6640006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:1226, start at 2024-04-26 19:12:56.737941139 +0800 CST m=+5.567492253 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:56.746 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:56.729 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:56.729 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ef72c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:1305, start at 2024-04-26 19:12:56.81392942 +0800 CST m=+5.579247446 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:14:56.821 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:12:56.779 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:02:56.779 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/kv_client_stream_reconnect/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/partition_table Starting Upstream PD... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:12:58 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/765cbc17-0d55-48c5-953a-3a796cc75ef0 {"id":"765cbc17-0d55-48c5-953a-3a796cc75ef0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b8197ef 765cbc17-0d55-48c5-953a-3a796cc75ef0 /tidb/cdc/default/default/upstream/7362132130863911661 {"id":7362132130863911661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/765cbc17-0d55-48c5-953a-3a796cc75ef0 {"id":"765cbc17-0d55-48c5-953a-3a796cc75ef0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b8197ef 765cbc17-0d55-48c5-953a-3a796cc75ef0 /tidb/cdc/default/default/upstream/7362132130863911661 {"id":7362132130863911661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/765cbc17-0d55-48c5-953a-3a796cc75ef0 {"id":"765cbc17-0d55-48c5-953a-3a796cc75ef0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b8197ef 765cbc17-0d55-48c5-953a-3a796cc75ef0 /tidb/cdc/default/default/upstream/7362132130863911661 {"id":7362132130863911661,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x 0 check diff failed 1-th time, retry later Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ***************** properties ***************** "mysql.port"="4000" "mysql.db"="consistent_replicate_ddl" "recordcount"="50" "mysql.host"="127.0.0.1" "readproportion"="0" "updateproportion"="0" "operationcount"="0" "threadcount"="10" "insertproportion"="0" "requestdistribution"="uniform" "readallfields"="true" "workload"="core" "scanproportion"="0" "fieldcount"="3" "mysql.user"="root" "dotransactions"="false" ********************************************** Run finished, takes 11.492472ms INSERT - Takes(s): 0.0, Count: 50, OPS: 8380.7, Avg(us): 2078, Min(us): 892, Max(us): 5783, 95th(us): 6000, 99th(us): 6000 + set +x + tso='449348889099370497 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348889099370497 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:13:00 CST 2024] <<<<<< START cdc server in default_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.default_value.26122614.out server --log-file /tmp/tidb_cdc_test/default_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/default_value/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:13:00 CST 2024] <<<<<< START cdc server in kv_client_stream_reconnect case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/kv/kvClientForceReconnect=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kv_client_stream_reconnect.25402542.out server --log-file /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kv_client_stream_reconnect/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_ddl.usertable1 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:01 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7774dc3c-59d8-48ee-b514-1199241e0d68 {"id":"7774dc3c-59d8-48ee-b514-1199241e0d68","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7a8ef5 7774dc3c-59d8-48ee-b514-1199241e0d68 /tidb/cdc/default/default/upstream/7362132128350310691 {"id":7362132128350310691,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7774dc3c-59d8-48ee-b514-1199241e0d68 {"id":"7774dc3c-59d8-48ee-b514-1199241e0d68","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7a8ef5 7774dc3c-59d8-48ee-b514-1199241e0d68 /tidb/cdc/default/default/upstream/7362132128350310691 {"id":7362132128350310691,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7774dc3c-59d8-48ee-b514-1199241e0d68 {"id":"7774dc3c-59d8-48ee-b514-1199241e0d68","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7a8ef5 7774dc3c-59d8-48ee-b514-1199241e0d68 /tidb/cdc/default/default/upstream/7362132128350310691 {"id":7362132128350310691,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.foreign_key.cli.2433.out cli changefeed create --start-ts=449348888516624388 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:01 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fe6df7a5-12a4-42a3-83bb-88a9de6c38f5 {"id":"fe6df7a5-12a4-42a3-83bb-88a9de6c38f5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ab20b fe6df7a5-12a4-42a3-83bb-88a9de6c38f5 /tidb/cdc/default/default/upstream/7362132126562150631 {"id":7362132126562150631,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fe6df7a5-12a4-42a3-83bb-88a9de6c38f5 {"id":"fe6df7a5-12a4-42a3-83bb-88a9de6c38f5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ab20b fe6df7a5-12a4-42a3-83bb-88a9de6c38f5 /tidb/cdc/default/default/upstream/7362132126562150631 {"id":7362132126562150631,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fe6df7a5-12a4-42a3-83bb-88a9de6c38f5 {"id":"fe6df7a5-12a4-42a3-83bb-88a9de6c38f5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ab20b fe6df7a5-12a4-42a3-83bb-88a9de6c38f5 /tidb/cdc/default/default/upstream/7362132126562150631 {"id":7362132126562150631,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_retry.cli.2671.out cli changefeed create --start-ts=449348888679677953 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' Create changefeed successfully! ID: 6054cfab-bf8c-483a-a596-938ff7f6d9c5 Info: {"upstream_id":7362132128350310691,"namespace":"default","id":"6054cfab-bf8c-483a-a596-938ff7f6d9c5","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:13:01.813540692+08:00","start_ts":449348888516624388,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348888516624388,"checkpoint_ts":449348888516624388,"checkpoint_time":"2024-04-26 19:12:56.336"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later [Fri Apr 26 19:13:01 CST 2024] <<<<<< START cdc server in processor_err_chan case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/sourcemanager/puller/ProcessorAddTableError=1*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_err_chan.26882690.out server --log-file /tmp/tidb_cdc_test/processor_err_chan/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_err_chan/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_err_chan/conf/server.toml --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: 6a619954-4c63-4d93-8755-ff4ac24824ef Info: {"upstream_id":7362132126562150631,"namespace":"default","id":"6a619954-4c63-4d93-8755-ff4ac24824ef","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:13:02.154815104+08:00","start_ts":449348888679677953,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348888679677953,"checkpoint_ts":449348888679677953,"checkpoint_time":"2024-04-26 19:12:56.958"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f43b4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:1144, start at 2024-04-26 19:13:01.705289721 +0800 CST m=+7.817720368 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:01.723 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:01.722 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:01.722 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:03 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b24b450e-6584-4200-abdb-87e026cbdc1f {"id":"b24b450e-6584-4200-abdb-87e026cbdc1f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b8b1af5 b24b450e-6584-4200-abdb-87e026cbdc1f /tidb/cdc/default/default/upstream/7362132143626777953 {"id":7362132143626777953,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b24b450e-6584-4200-abdb-87e026cbdc1f {"id":"b24b450e-6584-4200-abdb-87e026cbdc1f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b8b1af5 b24b450e-6584-4200-abdb-87e026cbdc1f /tidb/cdc/default/default/upstream/7362132143626777953 {"id":7362132143626777953,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b24b450e-6584-4200-abdb-87e026cbdc1f {"id":"b24b450e-6584-4200-abdb-87e026cbdc1f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b8b1af5 b24b450e-6584-4200-abdb-87e026cbdc1f /tidb/cdc/default/default/upstream/7362132143626777953 {"id":7362132143626777953,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.default_value.cli.2658.out cli changefeed create --start-ts=449348889099370497 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: cc5737bf-1a0d-45fc-b5d6-e340b185bc99 Info: {"upstream_id":7362132143626777953,"namespace":"default","id":"cc5737bf-1a0d-45fc-b5d6-e340b185bc99","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:13:03.51750704+08:00","start_ts":449348889099370497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348889099370497,"checkpoint_ts":449348889099370497,"checkpoint_time":"2024-04-26 19:12:58.559"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x check diff successfully check_safepoint_forward http://127.0.0.1:2379 7362132130863911661 449348889299910667 449348889221267457 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x table consistent_replicate_ddl.usertable1 exists table consistent_replicate_ddl.usertable2 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f43b4000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:1144, start at 2024-04-26 19:13:01.705289721 +0800 CST m=+7.817720368 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:01.723 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:01.722 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:01.722 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f458c0008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:1178, start at 2024-04-26 19:13:01.814648046 +0800 CST m=+7.744958305 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:01.833 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:01.846 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:01.846 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:04 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d3c1c29-0b38-4af9-a558-748134b16b0d {"id":"4d3c1c29-0b38-4af9-a558-748134b16b0d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b91d2f6 4d3c1c29-0b38-4af9-a558-748134b16b0d /tidb/cdc/default/default/upstream/7362132145476011700 {"id":7362132145476011700,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d3c1c29-0b38-4af9-a558-748134b16b0d {"id":"4d3c1c29-0b38-4af9-a558-748134b16b0d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b91d2f6 4d3c1c29-0b38-4af9-a558-748134b16b0d /tidb/cdc/default/default/upstream/7362132145476011700 {"id":7362132145476011700,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4d3c1c29-0b38-4af9-a558-748134b16b0d {"id":"4d3c1c29-0b38-4af9-a558-748134b16b0d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b91d2f6 4d3c1c29-0b38-4af9-a558-748134b16b0d /tidb/cdc/default/default/upstream/7362132145476011700 {"id":7362132145476011700,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:04 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c469e355-6838-40ed-b7eb-993ca1f3968a {"id":"c469e355-6838-40ed-b7eb-993ca1f3968a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7cb973 c469e355-6838-40ed-b7eb-993ca1f3968a /tidb/cdc/default/default/upstream/7362132129764618800 {"id":7362132129764618800,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c469e355-6838-40ed-b7eb-993ca1f3968a {"id":"c469e355-6838-40ed-b7eb-993ca1f3968a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7cb973 c469e355-6838-40ed-b7eb-993ca1f3968a /tidb/cdc/default/default/upstream/7362132129764618800 {"id":7362132129764618800,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c469e355-6838-40ed-b7eb-993ca1f3968a {"id":"c469e355-6838-40ed-b7eb-993ca1f3968a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7cb973 c469e355-6838-40ed-b7eb-993ca1f3968a /tidb/cdc/default/default/upstream/7362132129764618800 {"id":7362132129764618800,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 80da1d80-093c-42fd-b412-26a8de7c6273 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=80da1d80-093c-42fd-b412-26a8de7c6273 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 80da1d80-093c-42fd-b412-26a8de7c6273 -s Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/charset_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/charset_gbk/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } run task successfully + info='{ "upstream_id": 7362132129764618800, "namespace": "default", "id": "80da1d80-093c-42fd-b412-26a8de7c6273", "state": "normal", "checkpoint_tso": 449348890611417096, "checkpoint_time": "2024-04-26 19:13:04.327", "error": null }' + echo '{ "upstream_id": 7362132129764618800, "namespace": "default", "id": "80da1d80-093c-42fd-b412-26a8de7c6273", "state": "normal", "checkpoint_tso": 449348890611417096, "checkpoint_time": "2024-04-26 19:13:04.327", "error": null }' { "upstream_id": 7362132129764618800, "namespace": "default", "id": "80da1d80-093c-42fd-b412-26a8de7c6273", "state": "normal", "checkpoint_tso": 449348890611417096, "checkpoint_time": "2024-04-26 19:13:04.327", "error": null } ++ echo '{' '"upstream_id":' 7362132129764618800, '"namespace":' '"default",' '"id":' '"80da1d80-093c-42fd-b412-26a8de7c6273",' '"state":' '"normal",' '"checkpoint_tso":' 449348890611417096, '"checkpoint_time":' '"2024-04-26' '19:13:04.327",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132129764618800, '"namespace":' '"default",' '"id":' '"80da1d80-093c-42fd-b412-26a8de7c6273",' '"state":' '"normal",' '"checkpoint_tso":' 449348890611417096, '"checkpoint_time":' '"2024-04-26' '19:13:04.327",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state http://127.0.0.1:2379 7c78bf78-9fcf-44aa-b140-2d6991fb98e1 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=7c78bf78-9fcf-44aa-b140-2d6991fb98e1 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 7c78bf78-9fcf-44aa-b140-2d6991fb98e1 -s + info='{ "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "stopped", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null }' + echo '{ "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "stopped", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null }' { "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "stopped", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null } ++ echo '{' '"upstream_id":' 7362132130863911661, '"namespace":' '"default",' '"id":' '"7c78bf78-9fcf-44aa-b140-2d6991fb98e1",' '"state":' '"stopped",' '"checkpoint_tso":' 449348890545094660, '"checkpoint_time":' '"2024-04-26' '19:13:04.074",' '"error":' null '}' ++ jq -r .state + set +x go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading golang.org/x/time v0.5.0 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading github.com/coreos/go-semver v0.3.1 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7362132130863911661, '"namespace":' '"default",' '"id":' '"7c78bf78-9fcf-44aa-b140-2d6991fb98e1",' '"state":' '"stopped",' '"checkpoint_tso":' 449348890545094660, '"checkpoint_time":' '"2024-04-26' '19:13:04.074",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7362132130863911661 check diff failed 1-th time, retry later go: downloading golang.org/x/sys v0.19.0 go: downloading google.golang.org/protobuf v1.33.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading golang.org/x/net v0.24.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f7774000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:1205, start at 2024-04-26 19:13:05.026072687 +0800 CST m=+7.723421762 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:05.043 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:05.039 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:05.039 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading golang.org/x/text v0.14.0 table consistent_replicate_ddl.usertable2 not exists for 2-th check, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 12.76 secs (263800707 bytes/sec) [Pipeline] { [Pipeline] cache [Fri Apr 26 19:13:06 CST 2024] <<<<<< START cdc server in charset_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.charset_gbk.24332435.out server --log-file /tmp/tidb_cdc_test/charset_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/charset_gbk/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/google/btree v1.1.2 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.7 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading cloud.google.com/go v0.112.0 go: downloading google.golang.org/api v0.162.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/golang/snappy v0.0.4 check diff failed 2-th time, retry later table consistent_replicate_ddl.usertable2 exists table consistent_replicate_ddl.usertable3 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f7774000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:1205, start at 2024-04-26 19:13:05.026072687 +0800 CST m=+7.723421762 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:05.043 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:05.039 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:05.039 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686f761c000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:1238, start at 2024-04-26 19:13:04.945431794 +0800 CST m=+7.526902033 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:04.963 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:04.953 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:04.953 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resolve_lock/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resolve_lock/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading cloud.google.com/go/compute v1.24.0 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/kr/pretty v0.3.1 go: downloading cloud.google.com/go/iam v1.1.6 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/rivo/uniseg v0.4.6 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da run task successfully check_changefeed_state http://127.0.0.1:2379 7c78bf78-9fcf-44aa-b140-2d6991fb98e1 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=7c78bf78-9fcf-44aa-b140-2d6991fb98e1 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 7c78bf78-9fcf-44aa-b140-2d6991fb98e1 -s go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 + info='{ "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "normal", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null }' + echo '{ "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "normal", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null }' { "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "normal", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null } ++ echo '{' '"upstream_id":' 7362132130863911661, '"namespace":' '"default",' '"id":' '"7c78bf78-9fcf-44aa-b140-2d6991fb98e1",' '"state":' '"normal",' '"checkpoint_tso":' 449348890545094660, '"checkpoint_time":' '"2024-04-26' '19:13:04.074",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132130863911661, '"namespace":' '"default",' '"id":' '"7c78bf78-9fcf-44aa-b140-2d6991fb98e1",' '"state":' '"normal",' '"checkpoint_tso":' 449348890545094660, '"checkpoint_time":' '"2024-04-26' '19:13:04.074",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_forward http://127.0.0.1:2379 7362132130863911661 449348890545094659 449348890545094660 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 table consistent_replicate_ddl.usertable3 not exists for 2-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... run task successfully check_changefeed_state http://127.0.0.1:2379 7c78bf78-9fcf-44aa-b140-2d6991fb98e1 stopped null + endpoints=http://127.0.0.1:2379 + changefeed_id=7c78bf78-9fcf-44aa-b140-2d6991fb98e1 + expected_state=stopped + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 7c78bf78-9fcf-44aa-b140-2d6991fb98e1 -s + info='{ "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "stopped", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null }' + echo '{ "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "stopped", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null }' { "upstream_id": 7362132130863911661, "namespace": "default", "id": "7c78bf78-9fcf-44aa-b140-2d6991fb98e1", "state": "stopped", "checkpoint_tso": 449348890545094660, "checkpoint_time": "2024-04-26 19:13:04.074", "error": null } ++ echo '{' '"upstream_id":' 7362132130863911661, '"namespace":' '"default",' '"id":' '"7c78bf78-9fcf-44aa-b140-2d6991fb98e1",' '"state":' '"stopped",' '"checkpoint_tso":' 449348890545094660, '"checkpoint_time":' '"2024-04-26' '19:13:04.074",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7362132130863911661, '"namespace":' '"default",' '"id":' '"7c78bf78-9fcf-44aa-b140-2d6991fb98e1",' '"state":' '"stopped",' '"checkpoint_tso":' 449348890545094660, '"checkpoint_time":' '"2024-04-26' '19:13:04.074",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_changefeed_state http://127.0.0.1:2379 7caaa62b-1ba8-4ecd-97c2-3921bd0221d2 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=7caaa62b-1ba8-4ecd-97c2-3921bd0221d2 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 7caaa62b-1ba8-4ecd-97c2-3921bd0221d2 -s + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2476.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:10 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a7009772-1ba0-47ed-ab19-749db69b545e {"id":"a7009772-1ba0-47ed-ab19-749db69b545e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b9beff6 a7009772-1ba0-47ed-ab19-749db69b545e /tidb/cdc/default/default/upstream/7362132158883882663 {"id":7362132158883882663,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a7009772-1ba0-47ed-ab19-749db69b545e {"id":"a7009772-1ba0-47ed-ab19-749db69b545e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b9beff6 a7009772-1ba0-47ed-ab19-749db69b545e /tidb/cdc/default/default/upstream/7362132158883882663 {"id":7362132158883882663,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a7009772-1ba0-47ed-ab19-749db69b545e {"id":"a7009772-1ba0-47ed-ab19-749db69b545e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b9beff6 a7009772-1ba0-47ed-ab19-749db69b545e /tidb/cdc/default/default/upstream/7362132158883882663 {"id":7362132158883882663,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x Create changefeed successfully! ID: 106d8fa5-0ef5-45d9-855b-312286c67a62 Info: {"upstream_id":7362132158883882663,"namespace":"default","id":"106d8fa5-0ef5-45d9-855b-312286c67a62","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:13:10.29427664+08:00","start_ts":449348891278311428,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348891278311428,"checkpoint_ts":449348891278311428,"checkpoint_time":"2024-04-26 19:13:06.871"} wait process cdc.test exit for 2-th time... + info='{ "upstream_id": 7362132130863911661, "namespace": "default", "id": "7caaa62b-1ba8-4ecd-97c2-3921bd0221d2", "state": "normal", "checkpoint_tso": 449348892196601861, "checkpoint_time": "2024-04-26 19:13:10.374", "error": null }' + echo '{ "upstream_id": 7362132130863911661, "namespace": "default", "id": "7caaa62b-1ba8-4ecd-97c2-3921bd0221d2", "state": "normal", "checkpoint_tso": 449348892196601861, "checkpoint_time": "2024-04-26 19:13:10.374", "error": null }' { "upstream_id": 7362132130863911661, "namespace": "default", "id": "7caaa62b-1ba8-4ecd-97c2-3921bd0221d2", "state": "normal", "checkpoint_tso": 449348892196601861, "checkpoint_time": "2024-04-26 19:13:10.374", "error": null } ++ echo '{' '"upstream_id":' 7362132130863911661, '"namespace":' '"default",' '"id":' '"7caaa62b-1ba8-4ecd-97c2-3921bd0221d2",' '"state":' '"normal",' '"checkpoint_tso":' 449348892196601861, '"checkpoint_time":' '"2024-04-26' '19:13:10.374",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132130863911661, '"namespace":' '"default",' '"id":' '"7caaa62b-1ba8-4ecd-97c2-3921bd0221d2",' '"state":' '"normal",' '"checkpoint_tso":' 449348892196601861, '"checkpoint_time":' '"2024-04-26' '19:13:10.374",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully check_safepoint_equal http://127.0.0.1:2379 7362132130863911661 wait process cdc.test exit for 3-th time... table consistent_replicate_ddl.usertable3 exists table consistent_replicate_ddl.usertable_bak not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:13:11 CST 2024] <<<<<< run test case processor_err_chan success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + tso='449348892226748420 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348892226748420 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:13:12 CST 2024] <<<<<< START cdc server in resolve_lock case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.24962498.out server --log-file /tmp/tidb_cdc_test/resolve_lock/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resolve_lock/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ff5d40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:8068, start at 2024-04-26 19:13:13.124581909 +0800 CST m=+7.898928942 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:13.145 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:13.127 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:13.127 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_ddl.usertable_bak not exists for 2-th check, retry later go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd table foreign_key.finish_mark not exists for 1-th check, retry later run task successfully table foreign_key.finish_mark not exists for 2-th check, retry later Changefeed remove successfully. ID: 7c78bf78-9fcf-44aa-b140-2d6991fb98e1 CheckpointTs: 449348890545094660 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 check_safepoint_forward http://127.0.0.1:2379 7362132130863911661 449348892196601860 449348890545094660 449348892196601861 run task successfully + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:15 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e {"id":"bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba9adf8 bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e /tidb/cdc/default/default/upstream/7362132182577012573 {"id":7362132182577012573,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e {"id":"bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba9adf8 bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e /tidb/cdc/default/default/upstream/7362132182577012573 {"id":7362132182577012573,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e {"id":"bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ba9adf8 bc1d2d4b-4d67-4deb-b38a-0435d2a99e6e /tidb/cdc/default/default/upstream/7362132182577012573 {"id":7362132182577012573,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resolve_lock.cli.2535.out cli changefeed create --start-ts=449348892226748420 --sink-uri=mysql://normal:123456@127.0.0.1:3306/tidb-txn-mode=pessimistic VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ff5d40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:8068, start at 2024-04-26 19:13:13.124581909 +0800 CST m=+7.898928942 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:13.145 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:13.127 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:13.127 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c686ff7c8000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-s46cl, pid:8109, start at 2024-04-26 19:13:13.22586262 +0800 CST m=+7.877404393 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:13.242 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:13.253 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:13.253 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/partition_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Changefeed remove successfully. ID: 7caaa62b-1ba8-4ecd-97c2-3921bd0221d2 CheckpointTs: 449348893428678666 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 check_safepoint_cleared http://127.0.0.1:2379 7362132130863911661 run task successfully table foreign_key.finish_mark not exists for 3-th check, retry later Create changefeed successfully! ID: 4f6bb3d1-e033-482d-8f04-1df0ae60f170 Info: {"upstream_id":7362132182577012573,"namespace":"default","id":"4f6bb3d1-e033-482d-8f04-1df0ae60f170","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/tidb-txn-mode=pessimistic","create_time":"2024-04-26T19:13:15.865827401+08:00","start_ts":449348892226748420,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348892226748420,"checkpoint_ts":449348892226748420,"checkpoint_time":"2024-04-26 19:13:10.489"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table consistent_replicate_ddl.usertable_bak exists wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 7.62 secs (441605966 bytes/sec) [Pipeline] { cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:13:17 CST 2024] <<<<<< run test case gc_safepoint success! >>>>>> [Pipeline] cache + set +x go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading go.uber.org/zap v1.27.0 go: downloading go.uber.org/atomic v1.11.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/golang/protobuf v1.5.4 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading google.golang.org/protobuf v1.33.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading golang.org/x/sys v0.19.0 table foreign_key.finish_mark not exists for 4-th check, retry later table charset_gbk_test0.t0 exists table charset_gbk_test0.t1 exists table charset_gbk_test1.t0 not exists for 1-th check, retry later go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading golang.org/x/net v0.24.0 go: downloading golang.org/x/text v0.14.0 table foreign_key.finish_mark not exists for 5-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.9345.out cli tso query --pd=http://127.0.0.1:2379 table charset_gbk_test1.t0 exists table test.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 1-th time... + set +x + tso='449348894648434689 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348894648434689 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:13:21 CST 2024] <<<<<< START cdc server in partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.93729374.out server --log-file /tmp/tidb_cdc_test/partition_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/partition_table/cdc_data --cluster-id default * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_reconstruct/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:13:22 CST 2024] <<<<<< START cdc server in consistent_replicate_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true);github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true)' + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_ddl.28022804.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_ddl/cdcconsistent_replicate_ddl.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_ddl/cdc_dataconsistent_replicate_ddl.server2 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.162.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/klauspost/compress v1.17.7 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading golang.org/x/tools v0.20.0 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading cloud.google.com/go v0.112.0 go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/vbauerster/mpb/v7 v7.5.3 go: downloading golang.org/x/term v0.19.0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading cloud.google.com/go/compute v1.24.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/Masterminds/semver v1.5.0 go: downloading k8s.io/api v0.28.6 go: downloading github.com/emirpasic/gods v1.18.1 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/rivo/uniseg v0.4.6 table foreign_key.finish_mark not exists for 6-th check, retry later go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading cloud.google.com/go/iam v1.1.6 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da table test.finish_mark not exists for 2-th check, retry later go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/DataDog/zstd v1.5.5 ***************** properties ***************** "mysql.port"="4000" "operationcount"="0" "insertproportion"="0" "workload"="core" "threadcount"="2" "dotransactions"="false" "updateproportion"="0" "mysql.user"="root" "readproportion"="0" "recordcount"="10" "scanproportion"="0" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "mysql.db"="testSync" "readallfields"="true" ********************************************** Run finished, takes 5.69615ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2916.7, Avg(us): 992, Min(us): 393, Max(us): 3451, 95th(us): 4000, 99th(us): 4000 go: downloading k8s.io/apimachinery v0.28.6 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading k8s.io/klog/v2 v2.120.1 go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd table foreign_key.finish_mark not exists for 7-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:24 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7 {"id":"f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bc8a7f9 f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7 /tidb/cdc/default/default/upstream/7362132214186669630 {"id":7362132214186669630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7 {"id":"f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bc8a7f9 f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7 /tidb/cdc/default/default/upstream/7362132214186669630 {"id":7362132214186669630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7 {"id":"f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1bc8a7f9 f848e3dd-fe61-4cbd-88cf-7cb7fdf4c8b7 /tidb/cdc/default/default/upstream/7362132214186669630 {"id":7362132214186669630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.partition_table.cli.9408.out cli changefeed create --start-ts=449348894648434689 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ start tidb cluster in /tmp/tidb_cdc_test/changefeed_reconstruct Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table test.finish_mark not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:25 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/d4729c03-2982-476c-8d45-33f60d2a8ec8 {UpstreamID:7362132132842926423 Namespace:default ID:d4729c03-2982-476c-8d45-33f60d2a8ec8 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:12:58.345326066 +0800 CST StartTs:449348889013387270 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00400ee10 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348889039339523} {CheckpointTs:449348895737118724 MinTableBarrierTs:449348895999262725 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348895737118724, checkpointTs: 449348895737118724, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348895737118724, checkpointTs: 449348895737118724, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47357208-03d1-45ed-ae6d-96d1f337431d {"id":"47357208-03d1-45ed-ae6d-96d1f337431d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ee2b5 47357208-03d1-45ed-ae6d-96d1f337431d /tidb/cdc/default/default/changefeed/info/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"upstream-id":7362132132842926423,"namespace":"default","changefeed-id":"d4729c03-2982-476c-8d45-33f60d2a8ec8","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:12:58.345326066+08:00","start-ts":449348889013387270,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348889039339523} /tidb/cdc/default/default/changefeed/status/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"checkpoint-ts":449348895737118724,"min-table-barrier-ts":449348895999262725,"admin-job-type":0} /tidb/cdc/default/default/task/position/47357208-03d1-45ed-ae6d-96d1f337431d/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132132842926423 {"id":7362132132842926423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/d4729c03-2982-476c-8d45-33f60d2a8ec8 {UpstreamID:7362132132842926423 Namespace:default ID:d4729c03-2982-476c-8d45-33f60d2a8ec8 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:12:58.345326066 +0800 CST StartTs:449348889013387270 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00400ee10 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348889039339523} {CheckpointTs:449348895737118724 MinTableBarrierTs:449348895999262725 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348895737118724, checkpointTs: 449348895737118724, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348895737118724, checkpointTs: 449348895737118724, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47357208-03d1-45ed-ae6d-96d1f337431d {"id":"47357208-03d1-45ed-ae6d-96d1f337431d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ee2b5 47357208-03d1-45ed-ae6d-96d1f337431d /tidb/cdc/default/default/changefeed/info/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"upstream-id":7362132132842926423,"namespace":"default","changefeed-id":"d4729c03-2982-476c-8d45-33f60d2a8ec8","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:12:58.345326066+08:00","start-ts":449348889013387270,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348889039339523} /tidb/cdc/default/default/changefeed/status/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"checkpoint-ts":449348895737118724,"min-table-barrier-ts":449348895999262725,"admin-job-type":0} /tidb/cdc/default/default/task/position/47357208-03d1-45ed-ae6d-96d1f337431d/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132132842926423 {"id":7362132132842926423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/d4729c03-2982-476c-8d45-33f60d2a8ec8 {UpstreamID:7362132132842926423 Namespace:default ID:d4729c03-2982-476c-8d45-33f60d2a8ec8 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:12:58.345326066 +0800 CST StartTs:449348889013387270 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00400ee10 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348889039339523} {CheckpointTs:449348895737118724 MinTableBarrierTs:449348895999262725 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348895737118724, checkpointTs: 449348895737118724, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348895737118724, checkpointTs: 449348895737118724, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449348895737118725, checkpointTs: 449348895737118724, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/47357208-03d1-45ed-ae6d-96d1f337431d {"id":"47357208-03d1-45ed-ae6d-96d1f337431d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1b7ee2b5 47357208-03d1-45ed-ae6d-96d1f337431d /tidb/cdc/default/default/changefeed/info/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"upstream-id":7362132132842926423,"namespace":"default","changefeed-id":"d4729c03-2982-476c-8d45-33f60d2a8ec8","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:12:58.345326066+08:00","start-ts":449348889013387270,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_ddl/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348889039339523} /tidb/cdc/default/default/changefeed/status/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"checkpoint-ts":449348895737118724,"min-table-barrier-ts":449348895999262725,"admin-job-type":0} /tidb/cdc/default/default/task/position/47357208-03d1-45ed-ae6d-96d1f337431d/d4729c03-2982-476c-8d45-33f60d2a8ec8 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132132842926423 {"id":7362132132842926423,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 893a29b6-9dc6-45b1-b5f3-9de3a3a25ac8 Info: {"upstream_id":7362132214186669630,"namespace":"default","id":"893a29b6-9dc6-45b1-b5f3-9de3a3a25ac8","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:13:25.353016505+08:00","start_ts":449348894648434689,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348894648434689,"checkpoint_ts":449348894648434689,"checkpoint_time":"2024-04-26 19:13:19.727"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table foreign_key.finish_mark not exists for 8-th check, retry later Verifying downstream PD is started... table test.finish_mark not exists for 4-th check, retry later + set +x table foreign_key.finish_mark exists Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff successfully table test.finish_mark exists check table exists success check diff successfully wait process cdc.test exit for 1-th time... go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_pause_resume/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:13:30 CST 2024] <<<<<< run test case foreign_key success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/changefeed_pause_resume Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:13:31 CST 2024] <<<<<< run test case charset_gbk success! >>>>>> Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 10.26 secs (328075588 bytes/sec) [Pipeline] { [Pipeline] cache Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table sink_retry.finish_mark_1 exists check diff successfully ***************** properties ***************** "scanproportion"="0" "readproportion"="0" "threadcount"="2" "requestdistribution"="uniform" "updateproportion"="0" "recordcount"="10" "mysql.db"="sink_retry" "mysql.user"="root" "workload"="core" "mysql.host"="127.0.0.1" "operationcount"="0" "readallfields"="true" "mysql.port"="4000" "insertproportion"="0" "dotransactions"="false" ********************************************** Run finished, takes 4.915685ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2888.0, Avg(us): 816, Min(us): 421, Max(us): 1910, 95th(us): 2000, 99th(us): 2000 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687150440017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:3891, start at 2024-04-26 19:13:35.273513352 +0800 CST m=+5.299295529 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:35.281 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:35.249 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:35.249 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687150440017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:3891, start at 2024-04-26 19:13:35.273513352 +0800 CST m=+5.299295529 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:35.281 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:35.249 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:35.249 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687151cc000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:3972, start at 2024-04-26 19:13:35.357990612 +0800 CST m=+5.333252901 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:35.365 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:35.347 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:35.347 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_reconstruct/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:13:40 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.52835285.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver1 --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_manager/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_puller_lag/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:43 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c26349c-0e51-4469-a190-06ecc59efd32 {"id":"8c26349c-0e51-4469-a190-06ecc59efd32","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c2b6af2 8c26349c-0e51-4469-a190-06ecc59efd32 /tidb/cdc/default/default/upstream/7362132316416401988 {"id":7362132316416401988,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c26349c-0e51-4469-a190-06ecc59efd32 {"id":"8c26349c-0e51-4469-a190-06ecc59efd32","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c2b6af2 8c26349c-0e51-4469-a190-06ecc59efd32 /tidb/cdc/default/default/upstream/7362132316416401988 {"id":7362132316416401988,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c26349c-0e51-4469-a190-06ecc59efd32 {"id":"8c26349c-0e51-4469-a190-06ecc59efd32","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c2b6af2 8c26349c-0e51-4469-a190-06ecc59efd32 /tidb/cdc/default/default/upstream/7362132316416401988 {"id":7362132316416401988,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6871a6580014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:4293, start at 2024-04-26 19:13:40.806746658 +0800 CST m=+5.522579286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:40.816 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:40.808 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:40.808 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6871a6580014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:4293, start at 2024-04-26 19:13:40.806746658 +0800 CST m=+5.522579286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:40.816 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:40.808 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:40.808 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6871a9400015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:4355, start at 2024-04-26 19:13:40.98196925 +0800 CST m=+5.638432940 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:40.988 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:40.995 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:40.995 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_pause_resume/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ***************** properties ***************** "recordcount"="50" "operationcount"="0" "workload"="core" "insertproportion"="0" "mysql.port"="4000" "updateproportion"="0" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "mysql.db"="changefeed_reconstruct" "threadcount"="4" "readallfields"="true" "scanproportion"="0" "mysql.user"="root" "dotransactions"="false" "readproportion"="0" ********************************************** Run finished, takes 16.115493ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3963.5, Avg(us): 1290, Min(us): 859, Max(us): 4141, 95th(us): 4000, 99th(us): 5000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_manager Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/ddl_puller_lag Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later [Fri Apr 26 19:13:45 CST 2024] <<<<<< START cdc server in changefeed_pause_resume case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_pause_resume.56095611.out server --log-file /tmp/tidb_cdc_test/changefeed_pause_resume/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_pause_resume/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully wait process 5288 exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:48 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9419de9b-eedd-49a0-a36f-1b5d0c07fb01 {"id":"9419de9b-eedd-49a0-a36f-1b5d0c07fb01","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c4219f4 9419de9b-eedd-49a0-a36f-1b5d0c07fb01 /tidb/cdc/default/default/upstream/7362132338895491687 {"id":7362132338895491687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9419de9b-eedd-49a0-a36f-1b5d0c07fb01 {"id":"9419de9b-eedd-49a0-a36f-1b5d0c07fb01","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c4219f4 9419de9b-eedd-49a0-a36f-1b5d0c07fb01 /tidb/cdc/default/default/upstream/7362132338895491687 {"id":7362132338895491687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9419de9b-eedd-49a0-a36f-1b5d0c07fb01 {"id":"9419de9b-eedd-49a0-a36f-1b5d0c07fb01","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c4219f4 9419de9b-eedd-49a0-a36f-1b5d0c07fb01 /tidb/cdc/default/default/upstream/7362132338895491687 {"id":7362132338895491687,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x wait process 5288 exit for 2-th time... table partition_table.t exists table partition_table.t1 exists table partition_table.t2 not exists for 1-th check, retry later table partition_table.t2 exists Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (5288) - No such process wait process 5288 exit for 3-th time... process 5288 already exit check_no_capture http://127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 26 19:13:49 CST 2024] <<<<<< START cdc server in changefeed_reconstruct case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_reconstruct.55165518.out server --log-file /tmp/tidb_cdc_test/changefeed_reconstruct/cdcserver2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_reconstruct/cdc_dataserver2 --cluster-id default --addr 127.0.0.1:8300 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table partition_table.finish_mark not exists for 1-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:13:47 CST 2024] <<<<<< run test case kv_client_stream_reconnect success! >>>>>> table changefeed_pause_resume.t1 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:13:52 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/557708de-d2ef-4b28-8f05-80196e1ec986 {UpstreamID:7362132316416401988 Namespace:default ID:557708de-d2ef-4b28-8f05-80196e1ec986 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:13:43.594983631 +0800 CST StartTs:449348900867014664 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a43200 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348900893229063} {CheckpointTs:449348901810733064 MinTableBarrierTs:449348903121453062 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348901810733064, checkpointTs: 449348901810733064, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/86003c67-401c-4538-92f0-8a675ef5a4d5 {"id":"86003c67-401c-4538-92f0-8a675ef5a4d5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c2b6b67 86003c67-401c-4538-92f0-8a675ef5a4d5 /tidb/cdc/default/default/changefeed/info/557708de-d2ef-4b28-8f05-80196e1ec986 {"upstream-id":7362132316416401988,"namespace":"default","changefeed-id":"557708de-d2ef-4b28-8f05-80196e1ec986","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:13:43.594983631+08:00","start-ts":449348900867014664,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348900893229063} /tidb/cdc/default/default/changefeed/status/557708de-d2ef-4b28-8f05-80196e1ec986 {"checkpoint-ts":449348901810733064,"min-table-barrier-ts":449348903121453062,"admin-job-type":0} /tidb/cdc/default/default/task/position/86003c67-401c-4538-92f0-8a675ef5a4d5/557708de-d2ef-4b28-8f05-80196e1ec986 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132316416401988 {"id":7362132316416401988,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/557708de-d2ef-4b28-8f05-80196e1ec986 {UpstreamID:7362132316416401988 Namespace:default ID:557708de-d2ef-4b28-8f05-80196e1ec986 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:13:43.594983631 +0800 CST StartTs:449348900867014664 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a43200 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348900893229063} {CheckpointTs:449348901810733064 MinTableBarrierTs:449348903121453062 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348901810733064, checkpointTs: 449348901810733064, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/86003c67-401c-4538-92f0-8a675ef5a4d5 {"id":"86003c67-401c-4538-92f0-8a675ef5a4d5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c2b6b67 86003c67-401c-4538-92f0-8a675ef5a4d5 /tidb/cdc/default/default/changefeed/info/557708de-d2ef-4b28-8f05-80196e1ec986 {"upstream-id":7362132316416401988,"namespace":"default","changefeed-id":"557708de-d2ef-4b28-8f05-80196e1ec986","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:13:43.594983631+08:00","start-ts":449348900867014664,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348900893229063} /tidb/cdc/default/default/changefeed/status/557708de-d2ef-4b28-8f05-80196e1ec986 {"checkpoint-ts":449348901810733064,"min-table-barrier-ts":449348903121453062,"admin-job-type":0} /tidb/cdc/default/default/task/position/86003c67-401c-4538-92f0-8a675ef5a4d5/557708de-d2ef-4b28-8f05-80196e1ec986 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132316416401988 {"id":7362132316416401988,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/557708de-d2ef-4b28-8f05-80196e1ec986 {UpstreamID:7362132316416401988 Namespace:default ID:557708de-d2ef-4b28-8f05-80196e1ec986 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:13:43.594983631 +0800 CST StartTs:449348900867014664 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003a43200 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348900893229063} {CheckpointTs:449348901810733064 MinTableBarrierTs:449348903121453062 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348901810733064, checkpointTs: 449348901810733064, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/86003c67-401c-4538-92f0-8a675ef5a4d5 {"id":"86003c67-401c-4538-92f0-8a675ef5a4d5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c2b6b67 86003c67-401c-4538-92f0-8a675ef5a4d5 /tidb/cdc/default/default/changefeed/info/557708de-d2ef-4b28-8f05-80196e1ec986 {"upstream-id":7362132316416401988,"namespace":"default","changefeed-id":"557708de-d2ef-4b28-8f05-80196e1ec986","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:13:43.594983631+08:00","start-ts":449348900867014664,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348900893229063} /tidb/cdc/default/default/changefeed/status/557708de-d2ef-4b28-8f05-80196e1ec986 {"checkpoint-ts":449348901810733064,"min-table-barrier-ts":449348903121453062,"admin-job-type":0} /tidb/cdc/default/default/task/position/86003c67-401c-4538-92f0-8a675ef5a4d5/557708de-d2ef-4b28-8f05-80196e1ec986 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132316416401988 {"id":7362132316416401988,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list --pd=http://127.0.0.1:2379 2>&1 | grep id table changefeed_pause_resume.t1 exists table changefeed_pause_resume.t2 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release "id": "86003c67-401c-4538-92f0-8a675ef5a4d5", "cluster-id": "default" run task successfully table changefeed_pause_resume.t2 exists table changefeed_pause_resume.t3 not exists for 1-th check, retry later capture_id: 86003c67-401c-4538-92f0-8a675ef5a4d5 check_processor_table_count http://127.0.0.1:2379 557708de-d2ef-4b28-8f05-80196e1ec986 86003c67-401c-4538-92f0-8a675ef5a4d5 1 run task successfully check_processor_table_count http://127.0.0.1:2379 557708de-d2ef-4b28-8f05-80196e1ec986 86003c67-401c-4538-92f0-8a675ef5a4d5 0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... table partition_table.finish_mark not exists for 4-th check, retry later table count 1 does equal to expected count 0 run task failed 1-th time, retry later Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_pause_resume.t3 exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/cdc/run.sh using Sink-Type: mysql... <<================= ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table partition_table.finish_mark not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check_processor_table_count http://127.0.0.1:2379 557708de-d2ef-4b28-8f05-80196e1ec986 86003c67-401c-4538-92f0-8a675ef5a4d5 0 run task successfully ***************** properties ***************** "mysql.host"="127.0.0.1" "mysql.port"="4000" "readallfields"="true" "mysql.user"="root" "dotransactions"="false" "threadcount"="4" "workload"="core" "readproportion"="0" "mysql.db"="changefeed_reconstruct" "requestdistribution"="uniform" "operationcount"="0" "scanproportion"="0" "insertproportion"="0" "recordcount"="50" "updateproportion"="0" ********************************************** Run finished, takes 19.702936ms INSERT - Takes(s): 0.0, Count: 48, OPS: 3381.1, Avg(us): 1575, Min(us): 1038, Max(us): 5697, 95th(us): 6000, 99th(us): 6000 table changefeed_reconstruct.usertable not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6872c480000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:3414, start at 2024-04-26 19:13:59.104599518 +0800 CST m=+7.623282747 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:59.123 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:59.121 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:59.121 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6872c480000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:3414, start at 2024-04-26 19:13:59.104599518 +0800 CST m=+7.623282747 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:59.123 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:59.121 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:59.121 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6872c490000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:3449, start at 2024-04-26 19:13:59.099049504 +0800 CST m=+7.501881544 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:15:59.114 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:13:59.127 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:03:59.127 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_manager/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_manager/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table partition_table.finish_mark not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 22.29 secs (150990897 bytes/sec) [Pipeline] { [Pipeline] cache check diff failed 1-th time, retry later table changefeed_reconstruct.usertable exists check diff failed 1-th time, retry later table partition_table.finish_mark not exists for 7-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:14:02 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.46834685.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 2-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... table partition_table.finish_mark not exists for 8-th check, retry later wait process cdc.test exit for 2-th time... The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:14:04 CST 2024] <<<<<< run test case changefeed_reconstruct success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:05 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/959b3bde-55d8-4460-b588-283d686877af {"id":"959b3bde-55d8-4460-b588-283d686877af","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b5ff4 959b3bde-55d8-4460-b588-283d686877af /tidb/cdc/default/default/upstream/7362132413935621369 {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/959b3bde-55d8-4460-b588-283d686877af {"id":"959b3bde-55d8-4460-b588-283d686877af","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b5ff4 959b3bde-55d8-4460-b588-283d686877af /tidb/cdc/default/default/upstream/7362132413935621369 {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/959b3bde-55d8-4460-b588-283d686877af {"id":"959b3bde-55d8-4460-b588-283d686877af","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b5ff4 959b3bde-55d8-4460-b588-283d686877af /tidb/cdc/default/default/upstream/7362132413935621369 {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.cli.4726.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ -c=ddl-manager Create changefeed successfully! ID: ddl-manager Info: {"upstream_id":7362132413935621369,"namespace":"default","id":"ddl-manager","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-04-26T19:14:05.96911673+08:00","start_ts":449348906705747975,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348906705747975,"checkpoint_ts":449348906705747975,"checkpoint_time":"2024-04-26 19:14:05.722"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table partition_table.finish_mark not exists for 9-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687319e8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:3391, start at 2024-04-26 19:14:04.560330582 +0800 CST m=+8.221684190 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:04.587 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:04.588 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:04.588 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687319e8000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:3391, start at 2024-04-26 19:14:04.560330582 +0800 CST m=+8.221684190 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:04.587 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:04.588 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:04.588 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68731ad0000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:3417, start at 2024-04-26 19:14:04.622300446 +0800 CST m=+8.151970442 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:04.642 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:04.646 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:04.646 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_puller_lag/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 4-th time, retry later + set +x table sink_retry.finish_mark_2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:14:06 CST 2024] <<<<<< run test case sink_retry success! >>>>>> table partition_table.finish_mark not exists for 10-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/cdc Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check diff failed 5-th time, retry later table partition_table.finish_mark not exists for 11-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.4622.out cli tso query --pd=http://127.0.0.1:2379 check diff successfully table partition_table.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process 4688 exit for 1-th time... wait process 4688 exit for 2-th time... wait process cdc.test exit for 2-th time... wait process 4688 exit for 3-th time... wait process cdc.test exit for 3-th time... + set +x + tso='449348908283068417 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348908283068417 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x wait process 4688 exit for 4-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:14:14 CST 2024] <<<<<< run test case partition_table success! >>>>>> [Fri Apr 26 19:14:14 CST 2024] <<<<<< START cdc server in ddl_puller_lag case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorDDLResolved=1*sleep(180000)' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.46554657.out server --log-file /tmp/tidb_cdc_test/ddl_puller_lag/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_puller_lag/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_capture/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process 4688 exit for 5-th time... wait process 4688 exit for 6-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... wait process 4688 exit for 7-th time... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 12.20 secs (275736405 bytes/sec) [Pipeline] { [Pipeline] cache wait process 4688 exit for 8-th time... check diff failed 1-th time, retry later wait process 4688 exit for 9-th time... start tidb cluster in /tmp/tidb_cdc_test/multi_capture Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (4688) - No such process wait process 4688 exit for 10-th time... process 4688 already exit [Fri Apr 26 19:14:17 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.47824784.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:17 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab2afdaf-e78f-4066-a623-9b47c0e7476b {"id":"ab2afdaf-e78f-4066-a623-9b47c0e7476b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7ba316 ab2afdaf-e78f-4066-a623-9b47c0e7476b /tidb/cdc/default/default/upstream/7362132409147759630 {"id":7362132409147759630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab2afdaf-e78f-4066-a623-9b47c0e7476b {"id":"ab2afdaf-e78f-4066-a623-9b47c0e7476b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7ba316 ab2afdaf-e78f-4066-a623-9b47c0e7476b /tidb/cdc/default/default/upstream/7362132409147759630 {"id":7362132409147759630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab2afdaf-e78f-4066-a623-9b47c0e7476b {"id":"ab2afdaf-e78f-4066-a623-9b47c0e7476b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7ba316 ab2afdaf-e78f-4066-a623-9b47c0e7476b /tidb/cdc/default/default/upstream/7362132409147759630 {"id":7362132409147759630,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_puller_lag.cli.4696.out cli changefeed create --start-ts=449348908283068417 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ check diff failed 2-th time, retry later Create changefeed successfully! ID: 9cb85ec3-40e8-493b-afc7-b89261daa9b9 Info: {"upstream_id":7362132409147759630,"namespace":"default","id":"9cb85ec3-40e8-493b-afc7-b89261daa9b9","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:14:18.331299816+08:00","start_ts":449348908283068417,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348908283068417,"checkpoint_ts":449348908283068417,"checkpoint_time":"2024-04-26 19:14:11.739"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_error/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 3-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/changefeed_error Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check diff failed 4-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:23 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362132413935621369 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:14:05.96911673 +0800 CST StartTs:449348906705747975 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002393a70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348906757914630} {CheckpointTs:449348907269357571 MinTableBarrierTs:449348907269357571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b4d5e509-250a-4bb4-803c-6eb673a94396 {"id":"b4d5e509-250a-4bb4-803c-6eb673a94396","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b619c b4d5e509-250a-4bb4-803c-6eb673a94396 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362132413935621369,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:14:05.96911673+08:00","start-ts":449348906705747975,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348906757914630} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348907269357571,"min-table-barrier-ts":449348907269357571,"admin-job-type":0} /tidb/cdc/default/default/task/position/b4d5e509-250a-4bb4-803c-6eb673a94396/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132413935621369 {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362132413935621369 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:14:05.96911673 +0800 CST StartTs:449348906705747975 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002393a70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348906757914630} {CheckpointTs:449348907269357571 MinTableBarrierTs:449348907269357571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b4d5e509-250a-4bb4-803c-6eb673a94396 {"id":"b4d5e509-250a-4bb4-803c-6eb673a94396","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b619c b4d5e509-250a-4bb4-803c-6eb673a94396 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362132413935621369,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:14:05.96911673+08:00","start-ts":449348906705747975,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348906757914630} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348907269357571,"min-table-barrier-ts":449348907269357571,"admin-job-type":0} /tidb/cdc/default/default/task/position/b4d5e509-250a-4bb4-803c-6eb673a94396/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132413935621369 {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362132413935621369 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:14:05.96911673 +0800 CST StartTs:449348906705747975 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002393a70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348906757914630} {CheckpointTs:449348907269357571 MinTableBarrierTs:449348907269357571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b4d5e509-250a-4bb4-803c-6eb673a94396 {"id":"b4d5e509-250a-4bb4-803c-6eb673a94396","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b619c b4d5e509-250a-4bb4-803c-6eb673a94396 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362132413935621369,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:14:05.96911673+08:00","start-ts":449348906705747975,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348906757914630} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348907269357571,"min-table-barrier-ts":449348907269357571,"admin-job-type":0} /tidb/cdc/default/default/task/position/b4d5e509-250a-4bb4-803c-6eb673a94396/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132413935621369 + grep -q 'etcd info' {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [Fri Apr 26 19:14:23 CST 2024] <<<<<< START cdc server in ddl_manager case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteDDLSlowly=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_manager.48304832.out server --log-file /tmp/tidb_cdc_test/ddl_manager/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_manager/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 6.99 secs (481220017 bytes/sec) [Pipeline] { [Pipeline] cache check diff failed 5-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687430340013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:4029, start at 2024-04-26 19:14:22.374512238 +0800 CST m=+7.334354193 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:22.381 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:22.349 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:22.349 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687430340013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:4029, start at 2024-04-26 19:14:22.374512238 +0800 CST m=+7.334354193 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:22.381 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:22.349 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:22.349 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687431b0000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:4096, start at 2024-04-26 19:14:22.456298505 +0800 CST m=+5.535775242 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:22.463 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:22.444 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:22.444 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cdc/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cdc/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff successfully \033[0;36m<<< Run all test success >>>\033[0m ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:14:27 CST 2024] <<<<<< START cdc server in cdc case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc.53585360.out server --log-file /tmp/tidb_cdc_test/cdc/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:28 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362132413935621369 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:14:05.96911673 +0800 CST StartTs:449348906705747975 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002393a70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348906757914630} {CheckpointTs:449348907269357571 MinTableBarrierTs:449348907269357571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b4d5e509-250a-4bb4-803c-6eb673a94396 {"id":"b4d5e509-250a-4bb4-803c-6eb673a94396","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b619c b4d5e509-250a-4bb4-803c-6eb673a94396 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362132413935621369,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:14:05.96911673+08:00","start-ts":449348906705747975,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348906757914630} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348907295571984,"min-table-barrier-ts":449348907295571984,"admin-job-type":0} /tidb/cdc/default/default/task/position/b4d5e509-250a-4bb4-803c-6eb673a94396/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132413935621369 {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362132413935621369 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:14:05.96911673 +0800 CST StartTs:449348906705747975 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002393a70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348906757914630} {CheckpointTs:449348907269357571 MinTableBarrierTs:449348907269357571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b4d5e509-250a-4bb4-803c-6eb673a94396 {"id":"b4d5e509-250a-4bb4-803c-6eb673a94396","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b619c b4d5e509-250a-4bb4-803c-6eb673a94396 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362132413935621369,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:14:05.96911673+08:00","start-ts":449348906705747975,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348906757914630} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348907295571984,"min-table-barrier-ts":449348907295571984,"admin-job-type":0} /tidb/cdc/default/default/task/position/b4d5e509-250a-4bb4-803c-6eb673a94396/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132413935621369 {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-manager {UpstreamID:7362132413935621369 Namespace:default ID:ddl-manager SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:14:05.96911673 +0800 CST StartTs:449348906705747975 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002393a70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348906757914630} {CheckpointTs:449348907269357571 MinTableBarrierTs:449348907269357571 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b4d5e509-250a-4bb4-803c-6eb673a94396 {"id":"b4d5e509-250a-4bb4-803c-6eb673a94396","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1c7b619c b4d5e509-250a-4bb4-803c-6eb673a94396 /tidb/cdc/default/default/changefeed/info/ddl-manager {"upstream-id":7362132413935621369,"namespace":"default","changefeed-id":"ddl-manager","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:14:05.96911673+08:00","start-ts":449348906705747975,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348906757914630} /tidb/cdc/default/default/changefeed/status/ddl-manager {"checkpoint-ts":449348907295571984,"min-table-barrier-ts":449348907295571984,"admin-job-type":0} /tidb/cdc/default/default/task/position/b4d5e509-250a-4bb4-803c-6eb673a94396/ddl-manager {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132413935621369 + grep -q 'etcd info' {"id":7362132413935621369,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table ddl_manager.finish_mark not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68746a340004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:6665, start at 2024-04-26 19:14:26.06671269 +0800 CST m=+5.262202580 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:26.074 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:26.061 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:26.061 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68746a340004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:6665, start at 2024-04-26 19:14:26.06671269 +0800 CST m=+5.262202580 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:26.074 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:26.061 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:26.061 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68746ad40002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-2wz4q, pid:6748, start at 2024-04-26 19:14:26.101940583 +0800 CST m=+5.231327257 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:26.108 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:26.101 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:26.101 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_capture/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_capture/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_capture/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 2-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:30 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7688ef1a-7674-4e85-b0e4-04cfe0e40a35 {"id":"7688ef1a-7674-4e85-b0e4-04cfe0e40a35","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cd651f9 7688ef1a-7674-4e85-b0e4-04cfe0e40a35 /tidb/cdc/default/default/upstream/7362132504082602861 {"id":7362132504082602861,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7688ef1a-7674-4e85-b0e4-04cfe0e40a35 {"id":"7688ef1a-7674-4e85-b0e4-04cfe0e40a35","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cd651f9 7688ef1a-7674-4e85-b0e4-04cfe0e40a35 /tidb/cdc/default/default/upstream/7362132504082602861 {"id":7362132504082602861,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7688ef1a-7674-4e85-b0e4-04cfe0e40a35 {"id":"7688ef1a-7674-4e85-b0e4-04cfe0e40a35","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cd651f9 7688ef1a-7674-4e85-b0e4-04cfe0e40a35 /tidb/cdc/default/default/upstream/7362132504082602861 {"id":7362132504082602861,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc.cli.5402.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 6977916d-5be5-4212-a627-34b62a99494e Info: {"upstream_id":7362132504082602861,"namespace":"default","id":"6977916d-5be5-4212-a627-34b62a99494e","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:14:30.734522695+08:00","start_ts":449348913227104262,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348913227104262,"checkpoint_ts":449348913227104262,"checkpoint_time":"2024-04-26 19:14:30.599"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later table testSync.usertable exists table testSync.simple1 exists table testSync.simple2 exists + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8118.out cli tso query --pd=http://127.0.0.1:2379 table ddl_manager.finish_mark not exists for 3-th check, retry later + set +x go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading google.golang.org/grpc v1.62.1 go: downloading github.com/coreos/go-semver v0.3.1 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/golang/protobuf v1.5.4 go: downloading golang.org/x/net v0.24.0 go: downloading golang.org/x/sys v0.19.0 go: downloading google.golang.org/protobuf v1.33.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 + set +x + tso='449348913440227329 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348913440227329 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x go: downloading golang.org/x/text v0.14.0 check diff failed 2-th time, retry later ***************** properties ***************** "updateproportion"="0" "dotransactions"="false" "readproportion"="0" "insertproportion"="0" "mysql.user"="root" "mysql.db"="multi_capture_1" "readallfields"="true" "mysql.port"="4000" "requestdistribution"="uniform" "workload"="core" "scanproportion"="0" "operationcount"="0" "recordcount"="10" "threadcount"="2" "mysql.host"="127.0.0.1" ********************************************** Run finished, takes 10.190098ms INSERT - Takes(s): 0.0, Count: 10, OPS: 2116.9, Avg(us): 1977, Min(us): 1007, Max(us): 5445, 95th(us): 6000, 99th(us): 6000 table ddl_manager.finish_mark not exists for 4-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6874de2c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:3773, start at 2024-04-26 19:14:33.501352384 +0800 CST m=+5.220044250 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:33.507 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:33.483 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:33.483 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/gorilla/mux v1.8.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/otiai10/copy v1.2.0 go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/tidwall/btree v1.7.0 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.7 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading cloud.google.com/go v0.112.0 go: downloading google.golang.org/api v0.162.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 ***************** properties ***************** "recordcount"="10" "workload"="core" "readproportion"="0" "readallfields"="true" "mysql.host"="127.0.0.1" "scanproportion"="0" "mysql.port"="4000" "operationcount"="0" "requestdistribution"="uniform" "threadcount"="2" "updateproportion"="0" "mysql.user"="root" "dotransactions"="false" "insertproportion"="0" "mysql.db"="multi_capture_2" ********************************************** Run finished, takes 12.130371ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1674.5, Avg(us): 2316, Min(us): 1182, Max(us): 6059, 95th(us): 7000, 99th(us): 7000 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading cloud.google.com/go/compute v1.24.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading cloud.google.com/go/iam v1.1.6 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 check diff failed 3-th time, retry later go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/rivo/uniseg v0.4.6 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da ***************** properties ***************** "readproportion"="0" "mysql.db"="multi_capture_3" "workload"="core" "mysql.user"="root" "dotransactions"="false" "requestdistribution"="uniform" "mysql.port"="4000" "mysql.host"="127.0.0.1" "threadcount"="2" "recordcount"="10" "updateproportion"="0" "insertproportion"="0" "scanproportion"="0" "readallfields"="true" "operationcount"="0" ********************************************** Run finished, takes 9.671955ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1955.3, Avg(us): 1838, Min(us): 901, Max(us): 4540, 95th(us): 5000, 99th(us): 5000 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6874de2c0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:3773, start at 2024-04-26 19:14:33.501352384 +0800 CST m=+5.220044250 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:33.507 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:33.483 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:33.483 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6874e13c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:3849, start at 2024-04-26 19:14:33.700640975 +0800 CST m=+5.355818208 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:16:33.707 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:14:33.679 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:04:33.679 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_error/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_error/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 5-th check, retry later ***************** properties ***************** "readproportion"="0" "scanproportion"="0" "mysql.port"="4000" "mysql.host"="127.0.0.1" "readallfields"="true" "insertproportion"="0" "mysql.db"="multi_capture_4" "workload"="core" "threadcount"="2" "recordcount"="10" "updateproportion"="0" "mysql.user"="root" "requestdistribution"="uniform" "operationcount"="0" "dotransactions"="false" ********************************************** Run finished, takes 10.648956ms INSERT - Takes(s): 0.0, Count: 10, OPS: 1908.0, Avg(us): 2070, Min(us): 1044, Max(us): 5393, 95th(us): 6000, 99th(us): 6000 [Fri Apr 26 19:14:37 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.82388240.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 6-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5175.out cli tso query --pd=http://127.0.0.1:2379 check diff failed 5-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:40 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:14:40 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.82828284.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 7-th check, retry later + set +x + tso='449348915412074497 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348915412074497 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "readproportion"="0" "mysql.host"="127.0.0.1" "readallfields"="true" "threadcount"="4" "insertproportion"="0" "mysql.port"="4000" "mysql.db"="changefeed_error" "operationcount"="0" "mysql.user"="root" "scanproportion"="0" "recordcount"="20" "updateproportion"="0" "dotransactions"="false" "requestdistribution"="uniform" "workload"="core" ********************************************** Run finished, takes 10.038286ms INSERT - Takes(s): 0.0, Count: 20, OPS: 3530.4, Avg(us): 1849, Min(us): 914, Max(us): 4430, 95th(us): 5000, 99th(us): 5000 [Fri Apr 26 19:14:40 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedNoRetryError=1*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.52365238.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading github.com/felixge/httpsnoop v1.0.4 go: downloading github.com/jmespath/go-jmespath v0.4.0 check diff successfully table ddl_manager.finish_mark not exists for 8-th check, retry later go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:43 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d94fa4d1-a54d-4984-b9d6-034b85287286 {"id":"d94fa4d1-a54d-4984-b9d6-034b85287286","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6017b d94fa4d1-a54d-4984-b9d6-034b85287286 /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d94fa4d1-a54d-4984-b9d6-034b85287286 {"id":"d94fa4d1-a54d-4984-b9d6-034b85287286","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6017b d94fa4d1-a54d-4984-b9d6-034b85287286 /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d94fa4d1-a54d-4984-b9d6-034b85287286 {"id":"d94fa4d1-a54d-4984-b9d6-034b85287286","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6017b d94fa4d1-a54d-4984-b9d6-034b85287286 /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:14:43 CST 2024] <<<<<< START cdc server in multi_capture case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.83348336.out server --log-file /tmp/tidb_cdc_test/multi_capture/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_capture/cdc_data3 --cluster-id default --addr 127.0.0.1:8303 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8303; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:43 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 {"id":"b8f96992-2129-410f-950d-19e2c1c4d070","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6108 b8f96992-2129-410f-950d-19e2c1c4d070 /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 {"id":"b8f96992-2129-410f-950d-19e2c1c4d070","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6108 b8f96992-2129-410f-950d-19e2c1c4d070 /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 {"id":"b8f96992-2129-410f-950d-19e2c1c4d070","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6108 b8f96992-2129-410f-950d-19e2c1c4d070 /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5291.out cli changefeed create --start-ts=449348915412074497 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error table ddl_manager.finish_mark not exists for 9-th check, retry later Create changefeed successfully! ID: changefeed-error Info: {"upstream_id":7362132573846167428,"namespace":"default","id":"changefeed-error","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:14:44.884438187+08:00","start_ts":449348915412074497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348915412074497,"checkpoint_ts":449348915412074497,"checkpoint_time":"2024-04-26 19:14:38.934"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table ddl_manager.finish_mark not exists for 10-th check, retry later check diff failed 1-th time, retry later + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error failed [CDC:ErrStartTsBeforeGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=failed + error_msg='[CDC:ErrStartTsBeforeGC]' + tls_dir='[CDC:ErrStartTsBeforeGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449348915412074497, "checkpoint_time": "2024-04-26 19:14:38.934", "error": { "time": "2024-04-26T19:14:44.898120435+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348915412074197 is earlier than or equal to GC safepoint at 449348915412074497" } }' + echo '{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449348915412074497, "checkpoint_time": "2024-04-26 19:14:38.934", "error": { "time": "2024-04-26T19:14:44.898120435+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348915412074197 is earlier than or equal to GC safepoint at 449348915412074497" } }' { "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error", "state": "failed", "checkpoint_tso": 449348915412074497, "checkpoint_time": "2024-04-26 19:14:38.934", "error": { "time": "2024-04-26T19:14:44.898120435+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348915412074197 is earlier than or equal to GC safepoint at 449348915412074497" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 449348915412074497, '"checkpoint_time":' '"2024-04-26' '19:14:38.934",' '"error":' '{' '"time":' '"2024-04-26T19:14:44.898120435+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 449348915412074197 is earlier than or equal to GC safepoint at '449348915412074497"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"failed",' '"checkpoint_tso":' 449348915412074497, '"checkpoint_time":' '"2024-04-26' '19:14:38.934",' '"error":' '{' '"time":' '"2024-04-26T19:14:44.898120435+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]fail' to create or maintain changefeed because start-ts 449348915412074197 is earlier than or equal to GC safepoint at '449348915412074497"' '}' '}' + message='[CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348915412074197 is earlier than or equal to GC safepoint at 449348915412074497' + [[ ! [CDC:ErrStartTsBeforeGC]fail to create or maintain changefeed because start-ts 449348915412074197 is earlier than or equal to GC safepoint at 449348915412074497 =~ \[CDC:ErrStartTsBeforeGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5359.out cli changefeed resume -c changefeed-error + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8303/debug/info * About to connect() to 127.0.0.1 port 8303 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8303 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8303 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:14:46 GMT < Content-Length: 1144 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8697313f-b851-4577-b2b6-ec9e48644ca3 {"id":"8697313f-b851-4577-b2b6-ec9e48644ca3","address":"127.0.0.1:8303","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d94fa4d1-a54d-4984-b9d6-034b85287286 {"id":"d94fa4d1-a54d-4984-b9d6-034b85287286","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6017b d94fa4d1-a54d-4984-b9d6-034b85287286 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6018c 8697313f-b851-4577-b2b6-ec9e48644ca3 /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8697313f-b851-4577-b2b6-ec9e48644ca3 {"id":"8697313f-b851-4577-b2b6-ec9e48644ca3","address":"127.0.0.1:8303","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d94fa4d1-a54d-4984-b9d6-034b85287286 {"id":"d94fa4d1-a54d-4984-b9d6-034b85287286","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6017b d94fa4d1-a54d-4984-b9d6-034b85287286 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6018c 8697313f-b851-4577-b2b6-ec9e48644ca3 /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/1c60090d-3a8d-4067-b37e-0a601eb0ee5a {"id":"1c60090d-3a8d-4067-b37e-0a601eb0ee5a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8697313f-b851-4577-b2b6-ec9e48644ca3 {"id":"8697313f-b851-4577-b2b6-ec9e48644ca3","address":"127.0.0.1:8303","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d94fa4d1-a54d-4984-b9d6-034b85287286 {"id":"d94fa4d1-a54d-4984-b9d6-034b85287286","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6014f 1c60090d-3a8d-4067-b37e-0a601eb0ee5a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6017b d94fa4d1-a54d-4984-b9d6-034b85287286 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1cf6018c 8697313f-b851-4577-b2b6-ec9e48644ca3 /tidb/cdc/default/default/upstream/7362132540098700675 {"id":7362132540098700675,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_capture.cli.8377.out cli changefeed create --start-ts=449348913440227329 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 Create changefeed successfully! ID: d0dde1da-acb5-4330-98bf-bc2deae4e4ec Info: {"upstream_id":7362132540098700675,"namespace":"default","id":"d0dde1da-acb5-4330-98bf-bc2deae4e4ec","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:14:46.721188135+08:00","start_ts":449348913440227329,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348913440227329,"checkpoint_ts":449348913440227329,"checkpoint_time":"2024-04-26 19:14:31.412"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 20.49 secs (164194202 bytes/sec) [Pipeline] { [Pipeline] cache + set +x table multi_capture_1.usertable not exists for 1-th check, retry later check diff failed 2-th time, retry later + set +x table changefeed_error.usertable not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 11-th check, retry later table multi_capture_1.usertable exists table multi_capture_2.usertable not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 12-th check, retry later check diff failed 3-th time, retry later table changefeed_error.usertable exists check diff failed 1-th time, retry later table multi_capture_2.usertable exists table multi_capture_3.usertable not exists for 1-th check, retry later table ddl_manager.finish_mark not exists for 13-th check, retry later check diff failed 4-th time, retry later check diff successfully ***************** properties ***************** "mysql.db"="changefeed_error" "recordcount"="20" "mysql.host"="127.0.0.1" "readallfields"="true" "mysql.user"="root" "operationcount"="0" "insertproportion"="0" "mysql.port"="4000" "readproportion"="0" "scanproportion"="0" "requestdistribution"="uniform" "dotransactions"="false" "threadcount"="4" "updateproportion"="0" "workload"="core" ********************************************** Run finished, takes 5.48655ms INSERT - Takes(s): 0.0, Count: 20, OPS: 4819.3, Avg(us): 879, Min(us): 516, Max(us): 1859, 95th(us): 2000, 99th(us): 2000 check diff successfully {"id":"b8f96992-2129-410f-950d-19e2c1c4d070","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 + [[ /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 + echo 'check failed' check failed + exit 1 run task failed 1-th time, retry later table multi_capture_3.usertable exists table multi_capture_4.usertable not exists for 1-th check, retry later check diff failed 5-th time, retry later Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 5.77 secs (583291458 bytes/sec) [Pipeline] { [Pipeline] cache check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 + [[ /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 + echo 'check failed' check failed + exit 1 run task failed 2-th time, retry later table ddl_manager.finish_mark not exists for 14-th check, retry later table multi_capture_4.usertable exists check diff successfully ***************** properties ***************** "mysql.host"="127.0.0.1" "recordcount"="20" "mysql.port"="4000" "workload"="core" "readproportion"="0" "updateproportion"="0" "mysql.db"="multi_capture_1" "requestdistribution"="uniform" "operationcount"="0" "scanproportion"="0" "readallfields"="true" "mysql.user"="root" "dotransactions"="false" "insertproportion"="0" "threadcount"="2" ********************************************** Run finished, takes 11.831077ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1922.3, Avg(us): 1088, Min(us): 562, Max(us): 1956, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "updateproportion"="0" "threadcount"="2" "readallfields"="true" "requestdistribution"="uniform" "operationcount"="0" "mysql.user"="root" "scanproportion"="0" "dotransactions"="false" "recordcount"="20" "insertproportion"="0" "mysql.db"="multi_capture_2" "readproportion"="0" "mysql.host"="127.0.0.1" "mysql.port"="4000" "workload"="core" ********************************************** Run finished, takes 12.7669ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1736.4, Avg(us): 1085, Min(us): 541, Max(us): 1924, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "insertproportion"="0" "mysql.db"="multi_capture_3" "requestdistribution"="uniform" "mysql.user"="root" "updateproportion"="0" "dotransactions"="false" "operationcount"="0" "mysql.host"="127.0.0.1" "scanproportion"="0" "mysql.port"="4000" "recordcount"="20" "readproportion"="0" "workload"="core" "readallfields"="true" "threadcount"="2" ********************************************** Run finished, takes 25.316781ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1315.3, Avg(us): 2470, Min(us): 764, Max(us): 10820, 95th(us): 11000, 99th(us): 11000 ***************** properties ***************** "requestdistribution"="uniform" "workload"="core" "mysql.host"="127.0.0.1" "mysql.port"="4000" "mysql.db"="multi_capture_4" "dotransactions"="false" "scanproportion"="0" "operationcount"="0" "readallfields"="true" "threadcount"="2" "readproportion"="0" "insertproportion"="0" "updateproportion"="0" "recordcount"="20" "mysql.user"="root" ********************************************** Run finished, takes 10.483873ms INSERT - Takes(s): 0.0, Count: 20, OPS: 2126.8, Avg(us): 1002, Min(us): 491, Max(us): 1740, 95th(us): 2000, 99th(us): 2000 check diff failed 1-th time, retry later check diff successfully table ddl_manager.finish_mark not exists for 15-th check, retry later check diff successfully wait process cdc.test exit for 1-th time... check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info=/tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 + [[ /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 =~ capture ]] + echo 'capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070' capture contains in etcd /tidb/cdc/default/__cdc_meta__/capture/b8f96992-2129-410f-950d-19e2c1c4d070 + echo 'check failed' check failed + exit 1 run task failed 3-th time, retry later table ddl_manager.finish_mark not exists for 16-th check, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:00 CST 2024] <<<<<< run test case multi_capture success! >>>>>> check diff failed 1-th time, retry later table ddl_manager.finish_mark not exists for 17-th check, retry later check diff failed 2-th time, retry later table ddl_manager.finish_mark not exists for 18-th check, retry later check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/capture' 'capture' + key_prefix=/tidb/cdc/default/__cdc_meta__/capture + message=capture ++ etcdctl get /tidb/cdc/default/__cdc_meta__/capture --prefix --keys-only + info= + [[ '' =~ capture ]] + echo 'check pass' check pass + exit 0 run task successfully check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' [2024/04/26 19:15:01.658 +08:00] [INFO] [main.go:99] ["running ddl test: 1 modifyColumnDefaultValueDDL2"] [2024/04/26 19:15:01.658 +08:00] [INFO] [main.go:99] ["running ddl test: 0 modifyColumnDefaultValueDDL1"] [2024/04/26 19:15:01.989 +08:00] [INFO] [main.go:178] ["1 insert success: 100"] [2024/04/26 19:15:02.176 +08:00] [INFO] [main.go:178] ["0 insert success: 100"] [2024/04/26 19:15:02.177 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLscc20a4ba_a5bd_4f6a_9735_abe579b96811"] [2024/04/26 19:15:02.183 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs9837fe1c_d1b6_410c_8b68_2e3e1070f245"] [2024/04/26 19:15:02.190 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs27748da5_0e9c_4423_b8a4_3d01a28c3786"] [2024/04/26 19:15:02.191 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs3e6a3def_421b_4c0a_be33_69f47507b9c8"] [2024/04/26 19:15:02.195 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs97377c4d_b8d4_4594_bb4e_3f56cc0da7e9"] [2024/04/26 19:15:02.203 +08:00] [INFO] [main.go:178] ["1 insert success: 100"] [2024/04/26 19:15:02.270 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs52580ba1_4cea_4093_a111_d8f74ad46925"] [2024/04/26 19:15:02.271 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs02c363d1_779c_4f33_b070_09bdb11f91e2"] [2024/04/26 19:15:02.280 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsd84ddfc6_6a7a_4f61_9da6_6b23b1120cef"] [2024/04/26 19:15:02.300 +08:00] [INFO] [main.go:178] ["0 insert success: 100"] [2024/04/26 19:15:02.885 +08:00] [INFO] [main.go:178] ["1 insert success: 200"] [2024/04/26 19:15:03.176 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:03.185 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:03.190 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:03.272 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:03.277 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:03.290 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:03.298 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:03.300 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:03.395 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:03.469 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:03.473 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:03.479 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:03.480 +08:00] [INFO] [main.go:178] ["1 insert success: 200"] [2024/04/26 19:15:03.493 +08:00] [INFO] [main.go:178] ["0 insert success: 200"] [2024/04/26 19:15:03.493 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:03.495 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:03.497 +08:00] [INFO] [main.go:199] ["0 delete success: 100"] [2024/04/26 19:15:03.501 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:03.581 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:03.699 +08:00] [INFO] [main.go:178] ["0 insert success: 200"] [2024/04/26 19:15:03.784 +08:00] [INFO] [main.go:199] ["0 delete success: 100"] [2024/04/26 19:15:03.895 +08:00] [INFO] [main.go:178] ["1 insert success: 300"] [2024/04/26 19:15:04.195 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:04.294 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:04.385 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:04.393 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:04.471 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:04.487 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:04.489 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:04.489 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:04.489 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:04.493 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:04.495 +08:00] [INFO] [main.go:178] ["1 insert success: 300"] [2024/04/26 19:15:04.495 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:04.495 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:04.581 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:04.581 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:04.586 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:04.671 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:04.981 +08:00] [INFO] [main.go:178] ["1 insert success: 400"] + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 26 19:15:04 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/NewChangefeedRetryError=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.56025604.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:15:05.169 +08:00] [INFO] [main.go:178] ["0 insert success: 300"] table ddl_manager.finish_mark not exists for 19-th check, retry later [2024/04/26 19:15:05.392 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:05.473 +08:00] [INFO] [main.go:178] ["0 insert success: 300"] [2024/04/26 19:15:05.487 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:05.497 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:05.570 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:05.586 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:05.597 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] check diff failed 3-th time, retry later [2024/04/26 19:15:05.600 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:05.600 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:05.675 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:05.675 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:05.676 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:05.678 +08:00] [INFO] [main.go:178] ["1 insert success: 400"] [2024/04/26 19:15:05.681 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:05.697 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:05.697 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:05.771 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:05.776 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:06.085 +08:00] [INFO] [main.go:178] ["1 insert success: 500"] [2024/04/26 19:15:06.478 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:06.677 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:06.677 +08:00] [INFO] [main.go:178] ["0 insert success: 400"] [2024/04/26 19:15:06.683 +08:00] [INFO] [main.go:199] ["0 delete success: 200"] [2024/04/26 19:15:06.687 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:06.692 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:06.772 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:06.785 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:06.785 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:06.797 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:06.797 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:06.797 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:06.800 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:06.870 +08:00] [INFO] [main.go:178] ["1 insert success: 500"] [2024/04/26 19:15:06.871 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:06.884 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:06.886 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:06.889 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:06.900 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:06.976 +08:00] [INFO] [main.go:178] ["0 insert success: 400"] Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 9.39 secs (358525165 bytes/sec) [Pipeline] { [Pipeline] sh [2024/04/26 19:15:06.982 +08:00] [INFO] [main.go:199] ["0 delete success: 200"] [2024/04/26 19:15:07.189 +08:00] [INFO] [main.go:178] ["1 insert success: 600"] [Pipeline] sh + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G14 Run cases: big_txn changefeed_finish force_replicate_table PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=24138b88-81d2-48f4-a278-56508b05ede3 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G14 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-cxw2c GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-cxw2c GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/big_txn/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff failed 4-th time, retry later [Pipeline] sh [2024/04/26 19:15:07.595 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:07.794 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G12 Run cases: many_pk_or_uk capture_session_done_during_task ddl_attributes PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=5ba2e30a-ec4b-406e-92d4-a31c81be816d BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G12 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-0wbvc GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-0wbvc ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/many_pk_or_uk/run.sh using Sink-Type: mysql... <<================= table ddl_manager.finish_mark not exists for 20-th check, retry later [Pipeline] sh [2024/04/26 19:15:07.876 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:07.877 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:07.882 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:07.891 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:07.969 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:07.976 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:07.977 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:07.977 +08:00] [INFO] [main.go:178] ["1 insert success: 600"] [2024/04/26 19:15:07.979 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:07.981 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:07.983 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:07.987 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:07.992 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:07.992 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:07.997 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:08.187 +08:00] [INFO] [main.go:178] ["0 insert success: 500"] [2024/04/26 19:15:08.195 +08:00] [INFO] [main.go:178] ["1 insert success: 700"] + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G16 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:08 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c184c016-a2db-4cb1-b537-810d2e5e69b0 {"id":"c184c016-a2db-4cb1-b537-810d2e5e69b0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6195 c184c016-a2db-4cb1-b537-810d2e5e69b0 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7362132573846167428,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:14:44.884438187+08:00","start-ts":449348915412074497,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-04-26T19:15:05.369282433+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348922332676102} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449348918937911300,"min-table-barrier-ts":449348918937911300,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c184c016-a2db-4cb1-b537-810d2e5e69b0 {"id":"c184c016-a2db-4cb1-b537-810d2e5e69b0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6195 c184c016-a2db-4cb1-b537-810d2e5e69b0 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7362132573846167428,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:14:44.884438187+08:00","start-ts":449348915412074497,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-04-26T19:15:05.369282433+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348922332676102} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449348918937911300,"min-table-barrier-ts":449348918937911300,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c184c016-a2db-4cb1-b537-810d2e5e69b0 {"id":"c184c016-a2db-4cb1-b537-810d2e5e69b0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6195 c184c016-a2db-4cb1-b537-810d2e5e69b0 /tidb/cdc/default/default/changefeed/info/changefeed-error {"upstream-id":7362132573846167428,"namespace":"default","changefeed-id":"changefeed-error","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:14:44.884438187+08:00","start-ts":449348915412074497,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"pending","error":{"time":"2024-04-26T19:15:05.369282433+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrOwnerUnknown","message":"failpoint injected retriable error"},"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348922332676102} /tidb/cdc/default/default/changefeed/status/changefeed-error {"checkpoint-ts":449348918937911300,"min-table-barrier-ts":449348918937911300,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error warning failpoint injected retriable error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error + expected_state=warning + error_msg=failpoint + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error -s + info='{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449348918937911300, "checkpoint_time": "2024-04-26 19:14:52.384", "error": { "time": "2024-04-26T19:15:05.369282433+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' + echo '{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449348918937911300, "checkpoint_time": "2024-04-26 19:14:52.384", "error": { "time": "2024-04-26T19:15:05.369282433+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } }' { "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error", "state": "warning", "checkpoint_tso": 449348918937911300, "checkpoint_time": "2024-04-26 19:14:52.384", "error": { "time": "2024-04-26T19:15:05.369282433+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrOwnerUnknown", "message": "failpoint injected retriable error" } } ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 449348918937911300, '"checkpoint_time":' '"2024-04-26' '19:14:52.384",' '"error":' '{' '"time":' '"2024-04-26T19:15:05.369282433+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-error",' '"state":' '"warning",' '"checkpoint_tso":' 449348918937911300, '"checkpoint_time":' '"2024-04-26' '19:14:52.384",' '"error":' '{' '"time":' '"2024-04-26T19:15:05.369282433+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrOwnerUnknown",' '"message":' '"failpoint' injected retriable 'error"' '}' '}' ++ jq -r .error.message + message='failpoint injected retriable error' + [[ ! failpoint injected retriable error =~ failpoint ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5692.out cli changefeed remove -c changefeed-error [Pipeline] sh [2024/04/26 19:15:08.488 +08:00] [INFO] [main.go:178] ["0 insert success: 500"] [Pipeline] sh [2024/04/26 19:15:08.774 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] Run cases: owner_resign processor_etcd_worker_delay sink_hang PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=e5d17c73-e793-4e61-a3f6-b6a5e4465632 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G16 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-9hrjs GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-9hrjs ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/owner_resign/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G18 Run cases: availability http_proxies sequence PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=dc0840c8-9f39-4f0a-8293-1dee801ed520 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G18 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-xm95t GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-xm95t ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/availability/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] sh [2024/04/26 19:15:08.975 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:08.979 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:08.983 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:08.993 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:09.071 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:09.075 +08:00] [INFO] [main.go:178] ["1 insert success: 700"] [2024/04/26 19:15:09.081 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:09.083 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:09.086 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G15 Run cases: new_ci_collation batch_add_table multi_rocks PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=2326867e-317f-473c-ae3f-c2c3a93212ce BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G15 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-2jdqt GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-2jdqt GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/new_ci_collation/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/04/26 19:15:09.173 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:09.173 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:09.179 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:09.179 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:09.181 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:09.187 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:09.190 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] Changefeed remove successfully. ID: changefeed-error CheckpointTs: 449348918937911300 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... [Pipeline] sh check diff failed 5-th time, retry later [2024/04/26 19:15:09.374 +08:00] [INFO] [main.go:178] ["1 insert success: 800"] + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G20 Run cases: tidb_mysql_test ddl_reentrant multi_cdc_cluster PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=9828cdb5-0878-416d-a82b-be44d31062c8 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G20 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-3dv99 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-3dv99 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/tidb_mysql_test/run.sh using Sink-Type: mysql... <<================= [Pipeline] sh table ddl_manager.finish_mark not exists for 21-th check, retry later + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G21 Run cases: bank kill_owner_with_ddl owner_remove_table_error PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=1bd36dfb-534b-4d50-9ce8-dc37aa5362e8 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G21 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-r4370 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-r4370 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/bank/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/04/26 19:15:09.888 +08:00] [INFO] [main.go:178] ["0 insert success: 600"] [2024/04/26 19:15:09.889 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:09.892 +08:00] [INFO] [main.go:199] ["0 delete success: 300"] [2024/04/26 19:15:10.071 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:10.077 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:10.081 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:10.086 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:10.088 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:10.091 +08:00] [INFO] [main.go:178] ["0 insert success: 600"] [2024/04/26 19:15:10.093 +08:00] [INFO] [main.go:178] ["1 insert success: 800"] [2024/04/26 19:15:10.095 +08:00] [INFO] [main.go:199] ["0 delete success: 300"] \033[0;36m<<< Run all test success >>>\033[0m [2024/04/26 19:15:10.169 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:10.171 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:10.185 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:10.186 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:10.191 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:10.191 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:10.193 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:10.193 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:10.271 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:10.273 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:10.370 +08:00] [INFO] [main.go:178] ["1 insert success: 900"] + set +x check_no_changefeed 127.0.0.1:2379 parse error: Invalid numeric literal at line 1, column 6 run task successfully start tidb cluster in /tmp/tidb_cdc_test/big_txn Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:15:10.973 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:11.072 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:11.086 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:11.089 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:11.089 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:11.174 +08:00] [INFO] [main.go:178] ["1 insert success: 900"] [2024/04/26 19:15:11.176 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:11.190 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:11.190 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] wait process cdc.test exit for 1-th time... [2024/04/26 19:15:11.278 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:11.282 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:11.285 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:11.286 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:11.287 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:11.288 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:11.295 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:11.371 +08:00] [INFO] [main.go:178] ["0 insert success: 700"] [2024/04/26 19:15:11.375 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:11.392 +08:00] [INFO] [main.go:178] ["1 insert success: 1000"] wait process cdc.test exit for 2-th time... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G13 Run cases: tiflash region_merge common_1 PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=139a8360-6423-4389-a5da-6695b2aa406e BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G13 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-vzs14 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-vzs14 ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/tiflash/run.sh using Sink-Type: mysql... <<================= [2024/04/26 19:15:11.580 +08:00] [INFO] [main.go:178] ["0 insert success: 700"] table ddl_manager.finish_mark not exists for 22-th check, retry later The 1 times to try to start tidb cluster... check diff failed 6-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 26 19:15:12 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedDDLError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.57715773.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/availability Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:15:12.079 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:12.085 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:12.178 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:12.183 +08:00] [INFO] [main.go:178] ["1 insert success: 1000"] [2024/04/26 19:15:12.277 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:12.280 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:12.285 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:12.296 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:12.296 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] start tidb cluster in /tmp/tidb_cdc_test/new_ci_collation Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:15:12.374 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:12.374 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:12.381 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:12.381 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:12.476 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:12.493 +08:00] [INFO] [main.go:178] ["1 insert success: 1100"] [2024/04/26 19:15:12.571 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:12.577 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:15:12.583 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/owner_resign Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... The 1 times to try to start tidb cluster... + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G17 Run cases: clustered_index processor_resolved_ts_fallback PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=db5bc9c1-8d86-463e-aa80-8c020cb69a6b BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4 HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT=tcp://10.233.0.1:443 KUBERNETES_PORT_443_TCP_PORT=443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G17 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-svhtl GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-svhtl ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4 GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/clustered_index/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/04/26 19:15:13.174 +08:00] [INFO] [main.go:178] ["0 insert success: 800"] [2024/04/26 19:15:13.180 +08:00] [INFO] [main.go:199] ["0 delete success: 400"] [2024/04/26 19:15:13.276 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:13.379 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:13.380 +08:00] [INFO] [main.go:178] ["1 insert success: 1100"] [2024/04/26 19:15:13.388 +08:00] [INFO] [main.go:178] ["0 insert success: 800"] [2024/04/26 19:15:13.393 +08:00] [INFO] [main.go:199] ["0 delete success: 400"] [2024/04/26 19:15:13.469 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:13.475 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:13.493 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:13.575 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:13.576 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:13.576 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:13.578 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:13.581 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:13.586 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:13.592 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] table ddl_manager.finish_mark not exists for 23-th check, retry later [2024/04/26 19:15:13.771 +08:00] [INFO] [main.go:178] ["1 insert success: 1200"] [2024/04/26 19:15:13.772 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:13.788 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:13.793 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:13.871 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] check diff successfully [2024/04/26 19:15:14.475 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:14.578 +08:00] [INFO] [main.go:178] ["1 insert success: 1200"] [2024/04/26 19:15:14.589 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:14.592 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:14.676 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:14.687 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] Verifying downstream PD is started... [2024/04/26 19:15:14.771 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:14.776 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:14.783 +08:00] [INFO] [main.go:178] ["0 insert success: 900"] [2024/04/26 19:15:14.790 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:14.791 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:14.872 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:14.875 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:14.875 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:14.893 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:14.974 +08:00] [INFO] [main.go:178] ["1 insert success: 1300"] [2024/04/26 19:15:14.979 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:14.982 +08:00] [INFO] [main.go:178] ["0 insert success: 900"] [2024/04/26 19:15:14.984 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:14.985 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:15 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cffbb994-2a30-45bf-b6c9-c8828e770c0c {"id":"cffbb994-2a30-45bf-b6c9-c8828e770c0c","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b61c8 cffbb994-2a30-45bf-b6c9-c8828e770c0c /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cffbb994-2a30-45bf-b6c9-c8828e770c0c {"id":"cffbb994-2a30-45bf-b6c9-c8828e770c0c","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b61c8 cffbb994-2a30-45bf-b6c9-c8828e770c0c /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cffbb994-2a30-45bf-b6c9-c8828e770c0c {"id":"cffbb994-2a30-45bf-b6c9-c8828e770c0c","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b61c8 cffbb994-2a30-45bf-b6c9-c8828e770c0c /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5818.out cli changefeed create --start-ts=449348915412074497 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error-1 Create changefeed successfully! ID: changefeed-error-1 Info: {"upstream_id":7362132573846167428,"namespace":"default","id":"changefeed-error-1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:15:15.556481906+08:00","start_ts":449348915412074497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348915412074497,"checkpoint_ts":449348915412074497,"checkpoint_time":"2024-04-26 19:14:38.934"} PASS table ddl_manager.finish_mark not exists for 24-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2024/04/26 19:15:15.685 +08:00] [INFO] [main.go:178] ["1 insert success: 1300"] [2024/04/26 19:15:15.774 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:15.775 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:15.779 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:15.790 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:15.791 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:15.875 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:15.890 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] start tidb cluster in /tmp/tidb_cdc_test/tidb_mysql_test Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:15:15.970 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:15.974 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:15.974 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:15.985 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:15.994 +08:00] [INFO] [main.go:178] ["1 insert success: 1400"] [2024/04/26 19:15:15.994 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:16.070 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:16.074 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] start tidb cluster in /tmp/tidb_cdc_test/tiflash Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:15:16.190 +08:00] [INFO] [main.go:178] ["0 insert success: 1000"] [2024/04/26 19:15:16.194 +08:00] [INFO] [main.go:199] ["0 delete success: 500"] [2024/04/26 19:15:16.384 +08:00] [INFO] [main.go:178] ["0 insert success: 1000"] [2024/04/26 19:15:16.388 +08:00] [INFO] [main.go:199] ["0 delete success: 500"] [2024/04/26 19:15:16.690 +08:00] [INFO] [main.go:178] ["1 insert success: 1400"] [2024/04/26 19:15:16.769 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:16.785 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:16.787 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:16.788 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:16.791 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:16.882 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... + set +x check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 794 0 --:--:-- --:--:-- --:--:-- 796 + info='{"state":"normal","resolved_ts":449348915412074497,"checkpoint_ts":449348915412074497}' + echo '{"state":"normal","resolved_ts":449348915412074497,"checkpoint_ts":449348915412074497}' {"state":"normal","resolved_ts":449348915412074497,"checkpoint_ts":449348915412074497} ++ echo '{"state":"normal","resolved_ts":449348915412074497,"checkpoint_ts":449348915412074497}' ++ jq -r .state [2024/04/26 19:15:16.981 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:16.988 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:16.991 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:16.992 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:16.992 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:17.071 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:17.076 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:17.078 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:17.087 +08:00] [INFO] [main.go:178] ["1 insert success: 1500"] Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + state=normal + [[ ! normal == \w\a\r\n\i\n\g ]] + echo 'changefeed state normal does not equal to warning' changefeed state normal does not equal to warning + exit 1 run task failed 1-th time, retry later table ddl_manager.finish_mark not exists for 25-th check, retry later [2024/04/26 19:15:17.770 +08:00] [INFO] [main.go:178] ["0 insert success: 1100"] [2024/04/26 19:15:17.793 +08:00] [INFO] [main.go:178] ["1 insert success: 1500"] start tidb cluster in /tmp/tidb_cdc_test/bank Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:15:17.871 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:17.878 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:17.881 +08:00] [INFO] [main.go:178] ["0 insert success: 1100"] [2024/04/26 19:15:17.887 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:17.990 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:17.995 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:18.075 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:18.075 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:18.079 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:18.084 +08:00] [INFO] [main.go:178] ["1 insert success: 1600"] [2024/04/26 19:15:18.085 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:18.088 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] start tidb cluster in /tmp/tidb_cdc_test/clustered_index Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2024/04/26 19:15:18.606 +08:00] [INFO] [main.go:178] ["1 insert success: 1600"] [2024/04/26 19:15:18.783 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:18.792 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:18.803 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:18.870 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:18.871 +08:00] [INFO] [main.go:178] ["0 insert success: 1200"] [2024/04/26 19:15:18.871 +08:00] [INFO] [main.go:178] ["1 insert success: 1700"] [2024/04/26 19:15:18.873 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:18.875 +08:00] [INFO] [main.go:199] ["0 delete success: 600"] [2024/04/26 19:15:18.878 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:18.899 +08:00] [INFO] [main.go:178] ["0 insert success: 1200"] [2024/04/26 19:15:18.902 +08:00] [INFO] [main.go:199] ["0 delete success: 600"] check_changefeed_status 127.0.0.1:8300 changefeed-error-1 warning last_warning ErrExecDDLFailed + endpoint=127.0.0.1:8300 + changefeed_id=changefeed-error-1 + expected_state=warning + field=last_warning + error_pattern=ErrExecDDLFailed ++ curl 127.0.0.1:8300/api/v2/changefeeds/changefeed-error-1/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 253 100 253 0 0 2326 0 --:--:-- --:--:-- --:--:-- 2342 + info='{"state":"warning","resolved_ts":449348915779076123,"checkpoint_ts":449348915779076123,"last_warning":{"time":"2024-04-26T19:15:17.631890242+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' + echo '{"state":"warning","resolved_ts":449348915779076123,"checkpoint_ts":449348915779076123,"last_warning":{"time":"2024-04-26T19:15:17.631890242+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}}' {"state":"warning","resolved_ts":449348915779076123,"checkpoint_ts":449348915779076123,"last_warning":{"time":"2024-04-26T19:15:17.631890242+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec DDL failed"}} ++ echo '{"state":"warning","resolved_ts":449348915779076123,"checkpoint_ts":449348915779076123,"last_warning":{"time":"2024-04-26T19:15:17.631890242+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .state Verifying downstream PD is started... [2024/04/26 19:15:19.374 +08:00] [INFO] [main.go:178] ["1 insert success: 1700"] table ddl_manager.finish_mark not exists for 26-th check, retry later + state=warning + [[ ! warning == \w\a\r\n\i\n\g ]] + [[ -z last_warning ]] ++ echo '{"state":"warning","resolved_ts":449348915779076123,"checkpoint_ts":449348915779076123,"last_warning":{"time":"2024-04-26T19:15:17.631890242+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrExecDDLFailed","message":"[CDC:ErrExecDDLFailed]exec' DDL 'failed"}}' ++ jq -r .last_warning.message + error_msg='[CDC:ErrExecDDLFailed]exec DDL failed' + [[ ! [CDC:ErrExecDDLFailed]exec DDL failed =~ ErrExecDDLFailed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5880.out cli changefeed remove -c changefeed-error-1 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:15:19.489 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs753916ee_e25f_4094_a64e_c9658c696235"] [2024/04/26 19:15:19.499 +08:00] [INFO] [main.go:178] ["73 insert success: 1600"] [2024/04/26 19:15:19.572 +08:00] [INFO] [main.go:178] ["73 insert success: 1600"] [2024/04/26 19:15:19.591 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/04/26 19:15:19.593 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/04/26 19:15:19.597 +08:00] [INFO] [main.go:178] ["1 insert success: 1800"] [2024/04/26 19:15:19.713 +08:00] [INFO] [main.go:178] ["0 insert success: 1300"] Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:19.979 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:19.983 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:19.988 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs06437270_775a_4bc7_8c59_3672fba5db90"] [2024/04/26 19:15:19.994 +08:00] [INFO] [main.go:178] ["73 insert success: 1700"] [2024/04/26 19:15:19.995 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs89b3916d_0283_4e27_b9ac_2e13df180768"] [2024/04/26 19:15:20.096 +08:00] [INFO] [main.go:178] ["73 insert success: 1700"] [2024/04/26 19:15:20.103 +08:00] [INFO] [main.go:178] ["72 insert success: 1700"] Changefeed remove successfully. ID: changefeed-error-1 CheckpointTs: 449348915779076123 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:15:20.171 +08:00] [INFO] [main.go:178] ["72 insert success: 1700"] [2024/04/26 19:15:20.181 +08:00] [INFO] [main.go:178] ["1 insert success: 1900"] coverage: 2.1% of statements in github.com/pingcap/tiflow/... check diff failed 2-th time, retry later [2024/04/26 19:15:20.490 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs3347762b_2f77_4282_a9e6_7d46d4ec0e09"] [2024/04/26 19:15:20.496 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs57f260a5_4348_451d_a765_f422cc96ee11"] [2024/04/26 19:15:20.500 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLse31d48a9_317b_4ff4_8bd8_529c09ec1c8a"] [2024/04/26 19:15:20.601 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:20.604 +08:00] [INFO] [main.go:178] ["0 insert success: 1400"] [2024/04/26 19:15:20.669 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:20.671 +08:00] [INFO] [main.go:199] ["0 delete success: 700"] [2024/04/26 19:15:20.683 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:20.685 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:20.690 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:20.769 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:20.883 +08:00] [INFO] [main.go:178] ["1 insert success: 2000"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6877b06c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:1231, start at 2024-04-26 19:15:19.708523316 +0800 CST m=+5.114916444 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:19.715 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:19.707 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:19.707 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6877b06c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:1231, start at 2024-04-26 19:15:19.708523316 +0800 CST m=+5.114916444 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:19.715 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:19.707 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:19.707 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6877b1140014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:1307, start at 2024-04-26 19:15:19.79841496 +0800 CST m=+5.130079127 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:19.805 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:19.799 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:19.799 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/big_txn/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/big_txn/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/big_txn/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:21.381 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x [2024/04/26 19:15:21.671 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:21.673 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:21.772 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 27-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:21.873 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:21.980 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:22.072 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] Starting Upstream TiDB... wait process cdc.test exit for 1-th time... [2024/04/26 19:15:22.174 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:22.271 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:22.276 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:22.279 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... [2024/04/26 19:15:22.483 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/many_pk_or_uk Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check diff failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit check_etcd_meta_not_exist '/tidb/cdc/default/__cdc_meta__/owner' 'owner' + key_prefix=/tidb/cdc/default/__cdc_meta__/owner + message=owner ++ etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix --keys-only + info= + [[ '' =~ owner ]] + echo 'check pass' check pass + exit 0 run task successfully [Fri Apr 26 19:15:23 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/txnutil/gc/InjectActualGCSafePoint=return(9223372036854775807)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.59275929.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:15:23.073 +08:00] [INFO] [main.go:178] ["1 insert success: 2100"] [2024/04/26 19:15:23.074 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsc791d142_3df0_42a5_8fee_45c28e220a4e"] [2024/04/26 19:15:23.077 +08:00] [INFO] [main.go:178] ["0 insert success: 1500"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:15:22 CST 2024] <<<<<< START cdc server in big_txn case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.25442546.out server --log-file /tmp/tidb_cdc_test/big_txn/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/big_txn/cdc_data --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:23.574 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs91825bd5_32b5_4bc4_885c_124d40c945b5"] [2024/04/26 19:15:23.672 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] table ddl_manager.finish_mark not exists for 28-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6877e708000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:1207, start at 2024-04-26 19:15:23.218806953 +0800 CST m=+5.510895286 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:23.226 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:23.202 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:23.202 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:23.871 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:23.874 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:23.970 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [Pipeline] withEnv [Pipeline] { [2024/04/26 19:15:24.071 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:24.078 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:24.078 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:24.275 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:24.277 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:24.282 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:24.371 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:24.469 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:24.686 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:24.688 +08:00] [INFO] [main.go:178] ["1 insert success: 2200"] [2024/04/26 19:15:24.693 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:24.883 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:24.891 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:24.895 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:24.984 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:24.994 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:24.997 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] check diff failed 4-th time, retry later [2024/04/26 19:15:25.071 +08:00] [INFO] [main.go:178] ["0 insert success: 1600"] [2024/04/26 19:15:25.075 +08:00] [INFO] [main.go:199] ["0 delete success: 800"] [2024/04/26 19:15:25.077 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:25.084 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:25.085 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:25.173 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:25.188 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:25.192 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:25.271 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6877d7840009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:1165, start at 2024-04-26 19:15:22.219567519 +0800 CST m=+5.264873523 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:22.227 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:22.209 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:22.209 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6877d7840009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:1165, start at 2024-04-26 19:15:22.219567519 +0800 CST m=+5.264873523 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:22.227 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:22.209 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:22.209 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6877d8540002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:1236, start at 2024-04-26 19:15:22.262207766 +0800 CST m=+5.230227010 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:22.269 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:22.261 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:22.261 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/new_ci_collation/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/new_ci_collation/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/new_ci_collation/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:25.293 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table ddl_manager.finish_mark not exists for 29-th check, retry later [2024/04/26 19:15:25.784 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:25.875 +08:00] [INFO] [main.go:178] ["1 insert success: 2300"] [2024/04/26 19:15:25.879 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:26 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a79f47eb-1008-48ae-946a-fd2fed916003 {"id":"a79f47eb-1008-48ae-946a-fd2fed916003","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dc827f1 a79f47eb-1008-48ae-946a-fd2fed916003 /tidb/cdc/default/default/upstream/7362132771672577269 {"id":7362132771672577269,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a79f47eb-1008-48ae-946a-fd2fed916003 {"id":"a79f47eb-1008-48ae-946a-fd2fed916003","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dc827f1 a79f47eb-1008-48ae-946a-fd2fed916003 /tidb/cdc/default/default/upstream/7362132771672577269 {"id":7362132771672577269,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a79f47eb-1008-48ae-946a-fd2fed916003 {"id":"a79f47eb-1008-48ae-946a-fd2fed916003","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dc827f1 a79f47eb-1008-48ae-946a-fd2fed916003 /tidb/cdc/default/default/upstream/7362132771672577269 {"id":7362132771672577269,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:15:26 CST 2024] <<<<<< START cdc server in big_txn case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.25912593.out server --log-file /tmp/tidb_cdc_test/big_txn/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/big_txn/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > [2024/04/26 19:15:26.074 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:26.087 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:26.087 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:26.186 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:26.186 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:26.276 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:26.282 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:26.283 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:26.284 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT + (( i++ )) + (( i <= 50 )) < HTTP/1.1 200 OK ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info bootstrapped True Bootstrap flag. Do not delete. * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < Date: Fri, 26 Apr 2024 11:15:26 GMT < Content-Length: 613 < HTTP/1.1 200 OK < Content-Type: text/plain; charset=utf-8 < Date: Fri, 26 Apr 2024 11:15:26 GMT < Content-Length: 681 tidb_server_version 179 Bootstrap version. Do not delete. < Content-Type: text/plain; charset=utf-8 < { [data not shown] < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. * Connection #0 to host 127.0.0.1 left intact + res=' ddl_table_version 3 DDL Table Version. Do not delete. *** processors info ***: tikv_gc_leader_uuid 63c6877e708000f Current GC worker leader UUID. (DO NOT EDIT) *** owner info ***: *** etcd info ***: *** processors info ***: /tidb/cdc/default/__cdc_meta__/capture/a96239dd-6fea-4dcb-a744-b8984876c5c9 tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:1207, start at 2024-04-26 19:15:23.218806953 +0800 CST m=+5.510895286 Host name and pid of current GC leader. (DO NOT EDIT) {"id":"a96239dd-6fea-4dcb-a744-b8984876c5c9","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} tikv_gc_leader_lease 20240426-19:17:23.226 +0800 Current GC worker leader lease. (DO NOT EDIT) *** etcd info ***: tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used /tidb/cdc/default/__cdc_meta__/meta/meta-version tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. /tidb/cdc/default/__cdc_meta__/capture/a79f47eb-1008-48ae-946a-fd2fed916003 1 tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count tikv_gc_last_run_time 20240426-19:15:23.202 +0800 The time when last GC starts. (DO NOT EDIT) {"id":"a79f47eb-1008-48ae-946a-fd2fed916003","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} tikv_gc_safe_point 20240426-19:05:23.202 +0800 All versions after safe point can be accessed. (DO NOT EDIT) 5 Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b621b tidb_server_version 179 Bootstrap version. Do not delete. /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 system_tz Asia/Shanghai TiDB Global System Timezone. a96239dd-6fea-4dcb-a744-b8984876c5c9 new_collation_enabled True If the new collations are enabled. Do not edit it. /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dc827f1 ddl_table_version 3 DDL Table Version. Do not delete. /tidb/cdc/default/default/upstream/7362132573846167428 tikv_gc_leader_uuid 63c6877e8800014 Current GC worker leader UUID. (DO NOT EDIT) a79f47eb-1008-48ae-946a-fd2fed916003 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' /tidb/cdc/default/default/upstream/7362132771672577269 tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:1279, start at 2024-04-26 19:15:23.334413879 +0800 CST m=+5.568352722 Host name and pid of current GC leader. (DO NOT EDIT) *** owner info ***: tikv_gc_leader_lease 20240426-19:17:23.345 +0800 Current GC worker leader lease. (DO NOT EDIT) {"id":7362132771672577269,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used *** processors info ***: tikv_gc_enable true Current GC enable status + echo ' *** owner info ***: tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:23.347 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:23.347 +0800 All versions after safe point can be accessed. (DO NOT EDIT) *** processors info ***: *** etcd info ***: Starting Upstream TiFlash... TiFlash *** etcd info ***: Release Version: v7.5.1-12-g9002cc34d /tidb/cdc/default/__cdc_meta__/capture/a96239dd-6fea-4dcb-a744-b8984876c5c9 {"id":"a96239dd-6fea-4dcb-a744-b8984876c5c9","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} Edition: Community /tidb/cdc/default/__cdc_meta__/capture/a79f47eb-1008-48ae-946a-fd2fed916003 {"id":"a79f47eb-1008-48ae-946a-fd2fed916003","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dc827f1 a79f47eb-1008-48ae-946a-fd2fed916003 UTC Build Time: 2024-04-18 07:24:48 /tidb/cdc/default/default/upstream/7362132771672577269 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count {"id":7362132771672577269,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto *** owner info ***: 5 *** processors info ***: *** etcd info ***: Profile: RELWITHDEBINFO /tidb/cdc/default/__cdc_meta__/capture/a79f47eb-1008-48ae-946a-fd2fed916003 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b621b {"id":"a79f47eb-1008-48ae-946a-fd2fed916003","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} Raft Proxy /tidb/cdc/default/__cdc_meta__/meta/meta-version Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD a96239dd-6fea-4dcb-a744-b8984876c5c9 UTC Build Time: 2024-04-18 07:28:40 1 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) /tidb/cdc/default/default/upstream/7362132573846167428 Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' Profile: release /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dc827f1 Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure *** owner info ***: Verifying Upstream TiFlash is started... a79f47eb-1008-48ae-946a-fd2fed916003 Logging trace to /tmp/tidb_cdc_test/availability/tiflash/log/server.log *** processors info ***: *** etcd info ***: /tidb/cdc/default/default/upstream/7362132771672577269 Logging errors to /tmp/tidb_cdc_test/availability/tiflash/log/error.log /tidb/cdc/default/__cdc_meta__/capture/a96239dd-6fea-4dcb-a744-b8984876c5c9 {"id":7362132771672577269,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x {"id":"a96239dd-6fea-4dcb-a744-b8984876c5c9","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 5 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.big_txn.cli.2619.out cli changefeed create '--sink-uri=mysql://normal:123456@127.0.0.1:3306?transaction-atomicity=none' /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b621b a96239dd-6fea-4dcb-a744-b8984876c5c9 /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.5973.out cli changefeed create --start-ts=449348915412074497 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-error-2 arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/availability/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/availability/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/availability/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/04/26 19:15:26.385 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:26.391 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:26.395 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:26.486 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:26.495 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] Create changefeed successfully! ID: changefeed-error-2 Info: {"upstream_id":7362132573846167428,"namespace":"default","id":"changefeed-error-2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:15:26.585595296+08:00","start_ts":449348915412074497,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348915412074497,"checkpoint_ts":449348915412074497,"checkpoint_time":"2024-04-26 19:14:38.934"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Create changefeed successfully! ID: de3bc799-043c-40c1-9fb6-04fd5356c4c4 Info: {"upstream_id":7362132771672577269,"namespace":"default","id":"de3bc799-043c-40c1-9fb6-04fd5356c4c4","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306?transaction-atomicity=none","create_time":"2024-04-26T19:15:26.499178936+08:00","start_ts":449348927843991557,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"transaction_atomicity":"none","encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348927843991557,"checkpoint_ts":449348927843991557,"checkpoint_time":"2024-04-26 19:15:26.358"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:26.593 +08:00] [INFO] [main.go:178] ["0 insert success: 1700"] [2024/04/26 19:15:26.772 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:26.787 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] check diff failed 5-th time, retry later [2024/04/26 19:15:26.870 +08:00] [INFO] [main.go:178] ["1 insert success: 2400"] [2024/04/26 19:15:26.982 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:27.174 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:27.276 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ____________________________________ *************************** 1. row *************************** primary_ts: 449348861415391233 secondary_ts: 449348864061997073 *************************** 2. row *************************** primary_ts: 449348869279711232 secondary_ts: 449348870078201857 *************************** 3. row *************************** primary_ts: 449348877144031232 secondary_ts: 449348877418233857 *************************** 4. row *************************** primary_ts: 449348885008351232 secondary_ts: 449348885282553857 *************************** 5. row *************************** primary_ts: 449348892872671232 secondary_ts: 449348893173088257 *************************** 6. row *************************** primary_ts: 449348900736991232 secondary_ts: 449348901535744001 *************************** 7. row *************************** primary_ts: 449348908601311232 secondary_ts: 449348908901728257 *************************** 8. row *************************** primary_ts: 449348916465631232 secondary_ts: 449348916792262657 *************************** 9. row *************************** primary_ts: 449348924329951232 secondary_ts: 449348924669689857 skip invalid syncpoint primary_ts: 449348861415391233, first_ddl_ts: 449348864810156040 check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:24 CST 2024] <<<<<< run test case syncpoint success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:27.284 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:27.384 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:27.476 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:27.484 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:27.576 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:27.580 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] table ddl_manager.finish_mark not exists for 30-th check, retry later [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [2024/04/26 19:15:27.681 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:27.686 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:27.690 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:27.786 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:27.794 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [Fri Apr 26 19:15:27 CST 2024] <<<<<< START cdc server in new_ci_collation case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.new_ci_collation.25882590.out server --log-file /tmp/tidb_cdc_test/new_ci_collation/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/new_ci_collation/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-error-2 failed [CDC:ErrSnapshotLostByGC] + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-error-2 + expected_state=failed + error_msg='[CDC:ErrSnapshotLostByGC]' + tls_dir='[CDC:ErrSnapshotLostByGC]' + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-error-2 -s + set +x [2024/04/26 19:15:27.988 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:28.080 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:28.083 +08:00] [INFO] [main.go:178] ["1 insert success: 2500"] + info='{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:15:26.59480528+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348915412074497 is earlier than or equal to GC safepoint at 9223372036854775807" } }' + echo '{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:15:26.59480528+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348915412074497 is earlier than or equal to GC safepoint at 9223372036854775807" } }' { "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-error-2", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:15:26.59480528+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrSnapshotLostByGC", "message": "[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348915412074497 is earlier than or equal to GC safepoint at 9223372036854775807" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-04-26T19:15:26.59480528+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348915412074497 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-04-26T19:15:26.59480528+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrSnapshotLostByGC",' '"message":' '"[CDC:ErrSnapshotLostByGC]fail' to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348915412074497 is earlier than or equal to GC safepoint at '9223372036854775807"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348915412074497 is earlier than or equal to GC safepoint at 9223372036854775807' + [[ ! [CDC:ErrSnapshotLostByGC]fail to create or maintain changefeed due to snapshot loss caused by GC. checkpoint-ts 449348915412074497 is earlier than or equal to GC safepoint at 9223372036854775807 =~ \[CDC:ErrSnapshotLostByGC] ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6046.out cli changefeed remove -c changefeed-error-2 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.cli.2520.out cli tso query --pd=http://127.0.0.1:2379 [2024/04/26 19:15:28.180 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:28.375 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:28.482 +08:00] [INFO] [main.go:178] ["0 insert success: 1800"] [2024/04/26 19:15:28.483 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:28.569 +08:00] [INFO] [main.go:199] ["0 delete success: 900"] [2024/04/26 19:15:28.569 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:28.583 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68782bc00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:1115, start at 2024-04-26 19:15:27.631939697 +0800 CST m=+5.404897094 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:27.641 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:27.600 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:27.600 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully [2024/04/26 19:15:28.684 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:28.785 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:28.885 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] ***************** properties ***************** "operationcount"="0" "updateproportion"="0" "workload"="core" "requestdistribution"="uniform" "mysql.host"="127.0.0.1" "mysql.user"="root" "scanproportion"="0" "mysql.port"="4000" "mysql.db"="big_txn" "dotransactions"="false" "fieldcount"="100" "readproportion"="0" "recordcount"="5000" "readallfields"="true" "threadcount"="1" "insertproportion"="0" ********************************************** [2024/04/26 19:15:28.972 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:29.081 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:29.084 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:29.089 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] Changefeed remove successfully. ID: changefeed-error-2 CheckpointTs: 0 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:29.272 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:29.283 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:29.384 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] container [Pipeline] { table ddl_manager.finish_mark not exists for 31-th check, retry later [2024/04/26 19:15:29.478 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:29.481 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:29.488 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:29.578 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:29.591 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:29.672 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:29.677 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:29.770 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68783bec0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4, pid:1170, start at 2024-04-26 19:15:28.655566427 +0800 CST m=+5.327351929 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:28.663 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:28.635 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:28.635 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:29.978 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:29.980 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:30.074 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:30.173 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:30.181 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687823480012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:1291, start at 2024-04-26 19:15:27.076511576 +0800 CST m=+5.303341064 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:27.085 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:27.058 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:27.058 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687823480012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:1291, start at 2024-04-26 19:15:27.076511576 +0800 CST m=+5.303341064 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:27.085 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:27.058 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:27.058 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687823b80014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:1367, start at 2024-04-26 19:15:27.132897431 +0800 CST m=+5.286930752 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:27.140 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:27.136 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:27.136 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/tidb_mysql_test/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + set +x + tso='449348928366706692 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348928366706692 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x run test case test_kill_owner [Fri Apr 26 19:15:30 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.25752577.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_owner.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68782bc00013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:1115, start at 2024-04-26 19:15:27.631939697 +0800 CST m=+5.404897094 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:27.641 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:27.600 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:27.600 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68782bac0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:1197, start at 2024-04-26 19:15:27.637186478 +0800 CST m=+5.352546123 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:27.646 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:27.645 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:27.645 +0800 All versions after safe point can be accessed. (DO NOT EDIT) + set +x [Pipeline] } < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:30 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbcdc130-1367-4cb8-bfc1-d20b45b8b23a {"id":"bbcdc130-1367-4cb8-bfc1-d20b45b8b23a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dcf61f2 bbcdc130-1367-4cb8-bfc1-d20b45b8b23a /tidb/cdc/default/default/upstream/7362132776565538820 {"id":7362132776565538820,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbcdc130-1367-4cb8-bfc1-d20b45b8b23a {"id":"bbcdc130-1367-4cb8-bfc1-d20b45b8b23a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dcf61f2 bbcdc130-1367-4cb8-bfc1-d20b45b8b23a /tidb/cdc/default/default/upstream/7362132776565538820 {"id":7362132776565538820,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/bbcdc130-1367-4cb8-bfc1-d20b45b8b23a {"id":"bbcdc130-1367-4cb8-bfc1-d20b45b8b23a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dcf61f2 bbcdc130-1367-4cb8-bfc1-d20b45b8b23a /tidb/cdc/default/default/upstream/7362132776565538820 {"id":7362132776565538820,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [2024/04/26 19:15:30.484 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:30.580 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:30.674 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/bank/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/bank/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/bank/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/bank/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/bank/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Create changefeed successfully! ID: 4c2bdfea-812d-4d4d-bde4-3b27868076d1 Info: {"upstream_id":7362132776565538820,"namespace":"default","id":"4c2bdfea-812d-4d4d-bde4-3b27868076d1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2024-04-26T19:15:30.678251591+08:00","start_ts":449348928106397700,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348928106397700,"checkpoint_ts":449348928106397700,"checkpoint_time":"2024-04-26 19:15:27.359"} [2024/04/26 19:15:30.777 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:30.781 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:30.790 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:30.879 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:30.879 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:30.891 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:30.893 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:30.977 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] wait process cdc.test exit for 1-th time... [2024/04/26 19:15:31.080 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:31.175 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:31.178 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:31.272 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:31.373 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687848780012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:1127, start at 2024-04-26 19:15:29.4805139 +0800 CST m=+8.448132181 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:29.522 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:29.538 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:29.538 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687848780012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:1127, start at 2024-04-26 19:15:29.4805139 +0800 CST m=+8.448132181 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:29.522 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:29.538 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:29.538 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68784c8c0008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:1162, start at 2024-04-26 19:15:29.724125088 +0800 CST m=+8.575407353 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:29.752 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:29.750 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:29.750 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/tiflash/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/tiflash/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/tiflash/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... [2024/04/26 19:15:31.680 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68783bec0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4, pid:1170, start at 2024-04-26 19:15:28.655566427 +0800 CST m=+5.327351929 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:28.663 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:28.635 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:28.635 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68783d8c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4, pid:1252, start at 2024-04-26 19:15:28.766861721 +0800 CST m=+5.336982191 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:28.777 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:28.739 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:28.739 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/clustered_index/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/clustered_index/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/clustered_index/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:31 CST 2024] <<<<<< START cdc server in changefeed_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/redo/ChangefeedNewRedoManagerError=2*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.60876089.out server --log-file /tmp/tidb_cdc_test/changefeed_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_error/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:15:31.876 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:31.888 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:31.987 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:31.998 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] table ddl_manager.finish_mark not exists for 32-th check, retry later [2024/04/26 19:15:32.073 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:32.093 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:32.171 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:32.176 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:32.182 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:32.187 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:32.289 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:32.292 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:32.294 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:32.382 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:32.383 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tidb_mysql_test.cli.2612.out cli tso query --pd=http://127.0.0.1:2379 [2024/04/26 19:15:32.685 +08:00] [INFO] [main.go:88] ["testGetDefaultValue take 31.026763298s"] [2024/04/26 19:15:32.774 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] check diff failed 1-th time, retry later [2024/04/26 19:15:32.980 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:33.073 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:33.086 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:33.178 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:33.272 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:33.277 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:33.281 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:33.291 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [Fri Apr 26 19:15:32 CST 2024] <<<<<< START cdc server in bank case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.25822584.out server --log-file /tmp/tidb_cdc_test/bank/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/bank/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:33 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec2d145d-e6db-47e6-98f8-1b2515b818e1 {"id":"ec2d145d-e6db-47e6-98f8-1b2515b818e1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2913 ec2d145d-e6db-47e6-98f8-1b2515b818e1 /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec2d145d-e6db-47e6-98f8-1b2515b818e1 {"id":"ec2d145d-e6db-47e6-98f8-1b2515b818e1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2913 ec2d145d-e6db-47e6-98f8-1b2515b818e1 /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ec2d145d-e6db-47e6-98f8-1b2515b818e1 {"id":"ec2d145d-e6db-47e6-98f8-1b2515b818e1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2913 ec2d145d-e6db-47e6-98f8-1b2515b818e1 /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.cli.2624.out cli changefeed create --start-ts=449348928366706692 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [2024/04/26 19:15:33.380 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:33.385 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:33.386 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:33.483 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:33.490 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:33.494 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:33.498 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] Create changefeed successfully! ID: 0b0cc1de-0400-407b-bc84-46d5c1c5413a Info: {"upstream_id":7362132776889089863,"namespace":"default","id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:15:33.644785793+08:00","start_ts":449348928366706692,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348928366706692,"checkpoint_ts":449348928366706692,"checkpoint_time":"2024-04-26 19:15:28.352"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68787bfc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:1231, start at 2024-04-26 19:15:32.765612133 +0800 CST m=+5.481112724 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:32.774 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:32.735 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:32.735 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:15:33.772 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:33.784 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] table ddl_manager.finish_mark not exists for 33-th check, retry later + set +x + tso='449348929429962753 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348929429962753 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:15:33 CST 2024] <<<<<< START cdc server in tidb_mysql_test case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tidb_mysql_test.26422644.out server --log-file /tmp/tidb_cdc_test/tidb_mysql_test/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tidb_mysql_test/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:15:33.885 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:33.891 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:33.902 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:34.074 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:34.079 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2391.out cli tso query --pd=http://127.0.0.1:2379 [Fri Apr 26 19:15:33 CST 2024] <<<<<< START cdc server in clustered_index case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.clustered_index.25822584.out server --log-file /tmp/tidb_cdc_test/clustered_index/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/clustered_index/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:15:34.086 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:34.274 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:34.375 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:34.478 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:34.482 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:34.493 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:34.587 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:34.595 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:34.599 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:34.797 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] check diff failed 2-th time, retry later + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2024/04/26 19:15:34.881 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:34.975 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:34.979 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:34.988 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:35.072 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:35.076 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:35.076 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:35 GMT < Content-Length: 681 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc68d4ad-29b8-40cd-a0b0-9842ea1573b1 {"id":"dc68d4ad-29b8-40cd-a0b0-9842ea1573b1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6253 dc68d4ad-29b8-40cd-a0b0-9842ea1573b1 /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc68d4ad-29b8-40cd-a0b0-9842ea1573b1 {"id":"dc68d4ad-29b8-40cd-a0b0-9842ea1573b1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6253 dc68d4ad-29b8-40cd-a0b0-9842ea1573b1 /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/dc68d4ad-29b8-40cd-a0b0-9842ea1573b1 {"id":"dc68d4ad-29b8-40cd-a0b0-9842ea1573b1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 6 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1d0b6253 dc68d4ad-29b8-40cd-a0b0-9842ea1573b1 /tidb/cdc/default/default/upstream/7362132573846167428 {"id":7362132573846167428,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6143.out cli changefeed create --start-ts=0 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-initialize-error [2024/04/26 19:15:35.176 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:35.177 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:35.289 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:35.299 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:35.300 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] Create changefeed successfully! ID: changefeed-initialize-error Info: {"upstream_id":7362132573846167428,"namespace":"default","id":"changefeed-initialize-error","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:15:35.390027293+08:00","start_ts":449348930183888898,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348930183888898,"checkpoint_ts":449348930183888898,"checkpoint_time":"2024-04-26 19:15:35.284"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/hang_sink_suicide/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:15:34 CST 2024] <<<<<< run test case hang_sink_suicide success! >>>>>> [2024/04/26 19:15:35.475 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:35.486 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:35.569 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68787bfc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:1231, start at 2024-04-26 19:15:32.765612133 +0800 CST m=+5.481112724 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:32.774 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:32.735 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:32.735 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68787d8c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:1315, start at 2024-04-26 19:15:32.839750606 +0800 CST m=+5.490132595 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:32.848 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:32.835 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:32.835 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/many_pk_or_uk/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_manager.finish_mark not exists for 34-th check, retry later [2024/04/26 19:15:35.688 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:35.974 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:35.978 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:35.981 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] + set +x + tso='449348929922793473 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348929922793473 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:15:35 CST 2024] <<<<<< START cdc server in tiflash case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.24142416.out server --log-file /tmp/tidb_cdc_test/tiflash/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/tiflash/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:15:36.178 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:36.183 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:36.274 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:36.382 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:36 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0772e07b-61bf-4208-b2da-42a1a1608487 {"id":"0772e07b-61bf-4208-b2da-42a1a1608487","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1de3db00 0772e07b-61bf-4208-b2da-42a1a1608487 /tidb/cdc/default/default/upstream/7362132798148386340 {"id":7362132798148386340,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0772e07b-61bf-4208-b2da-42a1a1608487 {"id":"0772e07b-61bf-4208-b2da-42a1a1608487","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1de3db00 0772e07b-61bf-4208-b2da-42a1a1608487 /tidb/cdc/default/default/upstream/7362132798148386340 {"id":7362132798148386340,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0772e07b-61bf-4208-b2da-42a1a1608487 {"id":"0772e07b-61bf-4208-b2da-42a1a1608487","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1de3db00 0772e07b-61bf-4208-b2da-42a1a1608487 /tidb/cdc/default/default/upstream/7362132798148386340 {"id":7362132798148386340,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.bank.cli.2627.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ Create changefeed successfully! ID: 19e5758a-df0c-45ff-a8d6-8b426c256d85 Info: {"upstream_id":7362132798148386340,"namespace":"default","id":"19e5758a-df0c-45ff-a8d6-8b426c256d85","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-04-26T19:15:36.466183638+08:00","start_ts":449348930463334402,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348930463334402,"checkpoint_ts":449348930463334402,"checkpoint_time":"2024-04-26 19:15:36.350"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] // cache [Pipeline] } [2024/04/26 19:15:36.491 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:36.495 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:36.505 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:36.577 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:36.601 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s [2024/04/26 19:15:36.680 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:36.681 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:36.772 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:36.780 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] "is-owner": true, run task successfully + info='{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348930183888898, "checkpoint_time": "2024-04-26 19:15:35.284", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348930183888898, "checkpoint_time": "2024-04-26 19:15:35.284", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348930183888898, "checkpoint_time": "2024-04-26 19:15:35.284", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 449348930183888898, '"checkpoint_time":' '"2024-04-26' '19:15:35.284",' '"error":' '{' '"time":' '"2024-04-26T19:15:35.571039445+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6878a6540002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:1117, start at 2024-04-26 19:15:35.455270768 +0800 CST m=+16.652415714 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:35.499 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:35.545 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:35.545 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6878a6540002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:1117, start at 2024-04-26 19:15:35.455270768 +0800 CST m=+16.652415714 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:35.499 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:35.545 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:35.545 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68785fd40005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:1146, start at 2024-04-26 19:15:30.950078848 +0800 CST m=+12.011465766 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:30.980 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:30.982 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:30.982 +0800 All versions after safe point can be accessed. (DO NOT EDIT) [2024/04/26 19:15:37.079 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:36 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4 {"id":"29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ddc12f6 29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4 /tidb/cdc/default/default/upstream/7362132796854535754 {"id":7362132796854535754,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4 {"id":"29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ddc12f6 29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4 /tidb/cdc/default/default/upstream/7362132796854535754 {"id":7362132796854535754,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4 {"id":"29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ddc12f6 29792e85-73a4-4dbd-8ff9-bdcd62a6f3e4 /tidb/cdc/default/default/upstream/7362132796854535754 {"id":7362132796854535754,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 142 0 0 100 142 0 1466 --:--:-- --:--:-- --:--:-- 1479 mysql_test start go: downloading github.com/maxshuang/mysql-tester v0.0.2 check diff failed 3-th time, retry later [2024/04/26 19:15:37.381 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:37.386 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] table new_ci_collation_test.t1 exists table new_ci_collation_test.t2 exists table new_ci_collation_test.t3 exists table new_ci_collation_test.t4 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:36 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/774102d8-5e5d-41f7-833f-ecd4de27da9e {"id":"774102d8-5e5d-41f7-833f-ecd4de27da9e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1de461f2 774102d8-5e5d-41f7-833f-ecd4de27da9e /tidb/cdc/default/default/upstream/7362132810047964699 {"id":7362132810047964699,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/774102d8-5e5d-41f7-833f-ecd4de27da9e {"id":"774102d8-5e5d-41f7-833f-ecd4de27da9e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1de461f2 774102d8-5e5d-41f7-833f-ecd4de27da9e /tidb/cdc/default/default/upstream/7362132810047964699 {"id":7362132810047964699,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/774102d8-5e5d-41f7-833f-ecd4de27da9e {"id":"774102d8-5e5d-41f7-833f-ecd4de27da9e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1de461f2 774102d8-5e5d-41f7-833f-ecd4de27da9e /tidb/cdc/default/default/upstream/7362132810047964699 {"id":7362132810047964699,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 3b977a84-dc76-4fda-9863-29174ffdc66d Info: {"upstream_id":7362132810047964699,"namespace":"default","id":"3b977a84-dc76-4fda-9863-29174ffdc66d","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:15:37.056433831+08:00","start_ts":449348929791197188,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348929791197188,"checkpoint_ts":449348929791197188,"checkpoint_time":"2024-04-26 19:15:33.786"} Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... [2024/04/26 19:15:37.482 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:37.575 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:37.595 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:37.678 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:37.683 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] table ddl_manager.finish_mark not exists for 35-th check, retry later + set +x + GO111MODULE=on + go run bank.go case.go -u 'root@tcp(127.0.0.1:4000)/bank' -d 'root@tcp(127.0.0.1:3306)/bank' --test-round=20000 -a 127.0.0.1:20080 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/spf13/cobra v1.8.0 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading go.uber.org/zap v1.27.0 go: downloading google.golang.org/grpc v1.62.1 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 [2024/04/26 19:15:37.771 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:37.869 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:37.912 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] Logging trace to /tmp/tidb_cdc_test/owner_resign/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_resign/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/owner_resign/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // cache [Pipeline] } go: downloading github.com/spf13/pflag v1.0.5 go: downloading go.uber.org/multierr v1.11.0 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.cli.2633.out cli tso query --pd=http://127.0.0.1:2379 [2024/04/26 19:15:37.983 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:38.077 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:38.097 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:38.169 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:38.177 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/04/26 19:15:38.210 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] owner pid: 2580 owner id ec2d145d-e6db-47e6-98f8-1b2515b818e1 [Fri Apr 26 19:15:37 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.27212723.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_owner.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading golang.org/x/sys v0.19.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading google.golang.org/protobuf v1.33.0 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 [2024/04/26 19:15:38.381 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:38.485 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsf221137e_9964_449c_aa50_5aaa7c45a639"] [2024/04/26 19:15:38.675 +08:00] [INFO] [main.go:178] ["73 insert success: 1600"] [2024/04/26 19:15:38.874 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:38.879 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs27b6f444_61ce_49bb_b3c5_4407d1889732"] [2024/04/26 19:15:39.087 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:39.182 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] check diff failed 4-th time, retry later table new_ci_collation_test.t4 exists table new_ci_collation_test.t5 not exists for 1-th check, retry later [2024/04/26 19:15:39.295 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:39.372 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:39.384 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:39.388 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLscd109397_d860_48b1_953e_3fce99679664"] [2024/04/26 19:15:39.472 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:39.495 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:39.496 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:39 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0e00929b-81be-4738-937f-8d69dff4a7ca {"id":"0e00929b-81be-4738-937f-8d69dff4a7ca","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ddcecfa 0e00929b-81be-4738-937f-8d69dff4a7ca /tidb/cdc/default/default/upstream/7362132794517576213 {"id":7362132794517576213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0e00929b-81be-4738-937f-8d69dff4a7ca {"id":"0e00929b-81be-4738-937f-8d69dff4a7ca","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ddcecfa 0e00929b-81be-4738-937f-8d69dff4a7ca /tidb/cdc/default/default/upstream/7362132794517576213 {"id":7362132794517576213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0e00929b-81be-4738-937f-8d69dff4a7ca {"id":"0e00929b-81be-4738-937f-8d69dff4a7ca","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ddcecfa 0e00929b-81be-4738-937f-8d69dff4a7ca /tidb/cdc/default/default/upstream/7362132794517576213 {"id":7362132794517576213,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.tiflash.cli.2456.out cli changefeed create --start-ts=449348929922793473 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ + set +x + tso='449348930878832641 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348930878832641 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:15:39 CST 2024] <<<<<< START cdc server in many_pk_or_uk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.26712673.out server --log-file /tmp/tidb_cdc_test/many_pk_or_uk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/many_pk_or_uk/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348930183888898, "checkpoint_time": "2024-04-26 19:15:35.284", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348930183888898, "checkpoint_time": "2024-04-26 19:15:35.284", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "warning", "checkpoint_tso": 449348930183888898, "checkpoint_time": "2024-04-26 19:15:35.284", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"warning",' '"checkpoint_tso":' 449348930183888898, '"checkpoint_time":' '"2024-04-26' '19:15:35.284",' '"error":' '{' '"time":' '"2024-04-26T19:15:35.571039445+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=warning + [[ ! warning == \n\o\r\m\a\l ]] + echo 'changefeed state warning does not equal to normal' changefeed state warning does not equal to normal + exit 1 run task failed 2-th time, retry later table ddl_manager.finish_mark not exists for 36-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/server_config_compatibility/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] stage [Pipeline] { (Test) go: downloading github.com/pingcap/errors v0.11.5-0.20211009033009-93128226aaa3 go: downloading github.com/sirupsen/logrus v1.8.1 go: downloading github.com/pingcap/tidb/parser v0.0.0-20211122065349-4d16aabf900b go: downloading github.com/go-sql-driver/mysql v1.6.0 go: downloading go.uber.org/atomic v1.9.0 go: downloading golang.org/x/sys v0.0.0-20211117180635-dee7805ff2e1 go: downloading go.uber.org/zap v1.19.1 go: downloading github.com/pingcap/log v0.0.0-20210906054005-afc726e70354 go: downloading golang.org/x/text v0.3.7 go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.0.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0 go: downloading go.uber.org/multierr v1.7.0 [Fri Apr 26 19:15:39 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.24092411.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:15:39.771 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs9efce3c6_9763_4982_b52c_b7f8ed4a3076"] [2024/04/26 19:15:39.777 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] Create changefeed successfully! ID: db8ab209-5415-4fd2-8ccf-80796fa266c1 Info: {"upstream_id":7362132794517576213,"namespace":"default","id":"db8ab209-5415-4fd2-8ccf-80796fa266c1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:15:39.788502426+08:00","start_ts":449348929922793473,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348929922793473,"checkpoint_ts":449348929922793473,"checkpoint_time":"2024-04-26 19:15:34.288"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2024/04/26 19:15:40.179 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:40.179 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:40.194 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:40.199 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [Pipeline] // dir [Pipeline] } [2024/04/26 19:15:40.373 +08:00] [INFO] [main.go:178] ["73 insert success: 1600"] [2024/04/26 19:15:40.386 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs1f89e468_855a_4a40_bde6_99c54d5cbfb2"] [2024/04/26 19:15:40.388 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:40.398 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:40.495 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:40.497 +08:00] [INFO] [main.go:178] ["72 insert success: 1700"] [2024/04/26 19:15:40.500 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:40.785 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:40.879 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:40.884 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:40.893 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:40.980 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs7dde3079_8845_4a0a_9199_b49ed5c11593"] check diff failed 5-th time, retry later INSERT - Takes(s): 10.0, Count: 3156, OPS: 315.8, Avg(us): 2886, Min(us): 1294, Max(us): 83287, 95th(us): 3000, 99th(us): 68000 table new_ci_collation_test.t5 exists check diff failed 1-th time, retry later [Pipeline] // dir [Pipeline] } [2024/04/26 19:15:41.291 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:40 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c5fdd0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348928812351507 MinTableBarrierTs:449348931630399489 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e371e6f-9c88-49c1-98f4-b23cdc544504 {"id":"4e371e6f-9c88-49c1-98f4-b23cdc544504","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ec2d145d-e6db-47e6-98f8-1b2515b818e1 {"id":"ec2d145d-e6db-47e6-98f8-1b2515b818e1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2913 ec2d145d-e6db-47e6-98f8-1b2515b818e1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2961 4e371e6f-9c88-49c1-98f4-b23cdc544504 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348928812351507,"min-table-barrier-ts":449348931630399489,"admin-job-type":0} /tidb/cdc/default/default/task/position/4e371e6f-9c88-49c1-98f4-b23cdc544504/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ec2d145d-e6db-47e6-98f8-1b2515b818e1/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c5fdd0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348928812351507 MinTableBarrierTs:449348931630399489 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e371e6f-9c88-49c1-98f4-b23cdc544504 {"id":"4e371e6f-9c88-49c1-98f4-b23cdc544504","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ec2d145d-e6db-47e6-98f8-1b2515b818e1 {"id":"ec2d145d-e6db-47e6-98f8-1b2515b818e1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2913 ec2d145d-e6db-47e6-98f8-1b2515b818e1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2961 4e371e6f-9c88-49c1-98f4-b23cdc544504 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348928812351507,"min-table-barrier-ts":449348931630399489,"admin-job-type":0} /tidb/cdc/default/default/task/position/4e371e6f-9c88-49c1-98f4-b23cdc544504/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ec2d145d-e6db-47e6-98f8-1b2515b818e1/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c5fdd0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348928812351507 MinTableBarrierTs:449348931630399489 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4e371e6f-9c88-49c1-98f4-b23cdc544504 {"id":"4e371e6f-9c88-49c1-98f4-b23cdc544504","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ec2d145d-e6db-47e6-98f8-1b2515b818e1 {"id":"ec2d145d-e6db-47e6-98f8-1b2515b818e1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2913 ec2d145d-e6db-47e6-98f8-1b2515b818e1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2961 4e371e6f-9c88-49c1-98f4-b23cdc544504 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348928812351507,"min-table-barrier-ts":449348931630399489,"admin-job-type":0} /tidb/cdc/default/default/task/position/4e371e6f-9c88-49c1-98f4-b23cdc544504/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ec2d145d-e6db-47e6-98f8-1b2515b818e1/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep -v "ec2d145d-e6db-47e6-98f8-1b2515b818e1" | grep -v cluster_id | grep id [2024/04/26 19:15:41.372 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:41.373 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:41.488 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:41.590 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:41.591 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [Pipeline] timeout Timeout set to expire in 40 min [Pipeline] { [2024/04/26 19:15:41.798 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] + set +x table cdc_tiflash_test.multi_data_type not exists for 1-th check, retry later [2024/04/26 19:15:41.880 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:41.880 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:41.887 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:41.898 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:41.901 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:41.910 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs0c60be8b_6234_42b3_8e44_478a4c4b02b7"] [2024/04/26 19:15:42.002 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:42.007 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:42.010 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] table ddl_manager.finish_mark not exists for 37-th check, retry later [Pipeline] // withCredentials [Pipeline] } [2024/04/26 19:15:42.183 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:42.372 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] "cluster-id": "default" "id": "4e371e6f-9c88-49c1-98f4-b23cdc544504", "cluster-id": "default" run task successfully [Pipeline] // withCredentials [Pipeline] } [2024/04/26 19:15:42.386 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] table clustered_index_test.t0 exists table clustered_index_test.t1 exists table clustered_index_test.t2 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:42 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f0411036-c851-489d-ac99-dffe07c2d7dc {"id":"f0411036-c851-489d-ac99-dffe07c2d7dc","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1df3cef7 f0411036-c851-489d-ac99-dffe07c2d7dc /tidb/cdc/default/default/upstream/7362132825619079254 {"id":7362132825619079254,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f0411036-c851-489d-ac99-dffe07c2d7dc {"id":"f0411036-c851-489d-ac99-dffe07c2d7dc","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1df3cef7 f0411036-c851-489d-ac99-dffe07c2d7dc /tidb/cdc/default/default/upstream/7362132825619079254 {"id":7362132825619079254,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f0411036-c851-489d-ac99-dffe07c2d7dc {"id":"f0411036-c851-489d-ac99-dffe07c2d7dc","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1df3cef7 f0411036-c851-489d-ac99-dffe07c2d7dc /tidb/cdc/default/default/upstream/7362132825619079254 {"id":7362132825619079254,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.many_pk_or_uk.cli.2730.out cli changefeed create --start-ts=449348930878832641 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [Pipeline] withCredentials [2024/04/26 19:15:42.886 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] Masking supported pattern matches of $TICDC_COVERALLS_TOKEN or $TICDC_CODECOV_TOKEN [Pipeline] // timeout [Pipeline] } Create changefeed successfully! ID: 8bfe4fb1-bd7a-4f51-b7c8-685a700d3923 Info: {"upstream_id":7362132825619079254,"namespace":"default","id":"8bfe4fb1-bd7a-4f51-b7c8-685a700d3923","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:15:42.876006015+08:00","start_ts":449348930878832641,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348930878832641,"checkpoint_ts":449348930878832641,"checkpoint_time":"2024-04-26 19:15:37.935"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] // timeout [Pipeline] } [2024/04/26 19:15:42.971 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:42.974 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:42.984 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:42.987 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:42.994 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:43.077 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:43.084 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:43.095 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:43.102 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:43.105 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:42 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f87aa33-dd9a-4a87-b81c-a6d30183bba2 {"id":"4f87aa33-dd9a-4a87-b81c-a6d30183bba2","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b0a 4f87aa33-dd9a-4a87-b81c-a6d30183bba2 /tidb/cdc/default/default/upstream/7362132787447678527 {"id":7362132787447678527,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f87aa33-dd9a-4a87-b81c-a6d30183bba2 {"id":"4f87aa33-dd9a-4a87-b81c-a6d30183bba2","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b0a 4f87aa33-dd9a-4a87-b81c-a6d30183bba2 /tidb/cdc/default/default/upstream/7362132787447678527 {"id":7362132787447678527,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f87aa33-dd9a-4a87-b81c-a6d30183bba2 {"id":"4f87aa33-dd9a-4a87-b81c-a6d30183bba2","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b0a 4f87aa33-dd9a-4a87-b81c-a6d30183bba2 /tidb/cdc/default/default/upstream/7362132787447678527 {"id":7362132787447678527,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc cli capture list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' [Pipeline] { [2024/04/26 19:15:43.274 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:43.383 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [Pipeline] // stage [Pipeline] } [Pipeline] // stage [Pipeline] } check diff successfully [2024/04/26 19:15:43.476 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs49c09232_0cd8_4199_bf87_2a8309dc0364"] [2024/04/26 19:15:43.482 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] check diff failed 2-th time, retry later [Pipeline] // container [Pipeline] } check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s + info='{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348932058218500, "checkpoint_time": "2024-04-26 19:15:42.434", "error": null }' + echo '{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348932058218500, "checkpoint_time": "2024-04-26 19:15:42.434", "error": null }' { "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348932058218500, "checkpoint_time": "2024-04-26 19:15:42.434", "error": null } ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449348932058218500, '"checkpoint_time":' '"2024-04-26' '19:15:42.434",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449348932058218500, '"checkpoint_time":' '"2024-04-26' '19:15:42.434",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6274.out cli changefeed pause -c changefeed-initialize-error 1 run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.cli.2460.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 [Pipeline] // container [Pipeline] } capture_id: 4e371e6f-9c88-49c1-98f4-b23cdc544504 [Pipeline] // withEnv PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Pipeline] } table ddl_manager.finish_mark not exists for 38-th check, retry later [Pipeline] // withEnv table clustered_index_test.t2 exists check table exists success check diff failed 1-th time, retry later [Pipeline] } [Pipeline] dir Running in /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow [Pipeline] { [Pipeline] cache Create changefeed successfully! ID: 8291dabe-e8f7-4a20-a12b-aca8243b981d Info: {"upstream_id":7362132787447678527,"namespace":"default","id":"8291dabe-e8f7-4a20-a12b-aca8243b981d","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:15:44.144463822+08:00","start_ts":449348932454055938,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348932454055938,"checkpoint_ts":449348932454055938,"checkpoint_time":"2024-04-26 19:15:43.944"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... wait process 2580 exit for 1-th time... + set +x go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/atomic v1.11.0 go: downloading go.uber.org/multierr v1.11.0 table cdc_tiflash_test.multi_data_type exists check diff failed 1-th time, retry later [2024/04/26 19:15:44.278 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:44.482 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:44.484 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:44.486 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:44.578 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:44.582 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading google.golang.org/grpc v1.62.1 [2024/04/26 19:15:44.598 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:44.671 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:44.674 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:44.679 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:44.681 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:44.694 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:44.772 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:44.782 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:44.782 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:44.782 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] go: downloading golang.org/x/net v0.24.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading golang.org/x/sys v0.19.0 go: downloading google.golang.org/protobuf v1.33.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 wait process 2580 exit for 2-th time... go: downloading golang.org/x/text v0.14.0 + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error stopped changefeed new redo manager injected error + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=stopped + error_msg=changefeed + tls_dir=error + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s [2024/04/26 19:15:45.378 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (2580) - No such process wait process 2580 exit for 3-th time... process 2580 already exit cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 |grep 4e371e6f-9c88-49c1-98f4-b23cdc544504 -A1 | grep '"is-owner": true' check diff failed 3-th time, retry later + info='{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449348932320362500, "checkpoint_time": "2024-04-26 19:15:43.434", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' + echo '{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449348932320362500, "checkpoint_time": "2024-04-26 19:15:43.434", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } }' { "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "stopped", "checkpoint_tso": 449348932320362500, "checkpoint_time": "2024-04-26 19:15:43.434", "error": { "time": "2024-04-26T19:15:35.571039445+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrProcessorUnknown", "message": "changefeed new redo manager injected error" } } ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 449348932320362500, '"checkpoint_time":' '"2024-04-26' '19:15:43.434",' '"error":' '{' '"time":' '"2024-04-26T19:15:35.571039445+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"stopped",' '"checkpoint_tso":' 449348932320362500, '"checkpoint_time":' '"2024-04-26' '19:15:43.434",' '"error":' '{' '"time":' '"2024-04-26T19:15:35.571039445+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrProcessorUnknown",' '"message":' '"changefeed' new redo manager injected 'error"' '}' '}' ++ jq -r .error.message + message='changefeed new redo manager injected error' + [[ ! changefeed new redo manager injected error =~ changefeed ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6335.out cli changefeed resume -c changefeed-initialize-error [2024/04/26 19:15:45.581 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:45.581 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:45.582 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:45.677 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:45.683 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:45.778 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:45.779 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:45.785 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:45.789 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:45.791 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:45.798 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:45.881 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:45.885 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:45.895 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:45.895 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] table ddl_manager.finish_mark not exists for 39-th check, retry later [2024/04/26 19:15:46.181 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... + set +x Run finished, takes 17.300923034s INSERT - Takes(s): 17.3, Count: 5000, OPS: 289.1, Avg(us): 3166, Min(us): 1294, Max(us): 84940, 95th(us): 3000, 99th(us): 69000 table big_txn.usertable exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' [2024/04/26 19:15:46.575 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:46.575 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:46.678 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/consistent_replicate_ddl/sync_diff/output" source-instances = ["tidb"] target-instance = "mysql" target-check-tables = ["consistent_replicate_ddl.usertable*","consistent_replicate_ddl.check*"] [data-sources] [data-sources.tidb] host = "127.0.0.1" port = 4000 user = "root" password = "" snapshot = "449348897808056349" [data-sources.mysql] host = "127.0.0.1" port = 3306 user = "root" password = "" [2024/04/26 19:15:34.531 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/04/26 19:15:34.531 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/04/26 19:15:34.531 +08:00] [INFO] [redo.go:163] ["apply redo log starts"] [checkpointTs=449348896143441932] [resolvedTs=449348897808056349] [2024/04/26 19:15:34.532 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:15:34.532 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:15:34.533 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.696715ms] [2024/04/26 19:15:34.533 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=1.715611ms] [2024/04/26 19:15:34.539 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:15:34.542 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/04/26 19:15:34.542 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/04/26 19:15:34.543 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:15:34.543 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:15:34.548 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ALLOW_INVALID_DATES%2CNO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:15:34.550 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/04/26 19:15:34.550 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348896143441926,\"CommitTs\":449348896143441932,\"Query\":\"DROP TABLE `consistent_replicate_ddl`.`usertable1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1\",\"tbl-id\":106,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":4,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:34.551 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348896143441926,\"CommitTs\":449348896143441932,\"Query\":\"DROP TABLE `consistent_replicate_ddl`.`usertable1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1\",\"tbl-id\":106,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":4,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:34.770 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="DROP TABLE `consistent_replicate_ddl`.`usertable1`"] [duration=219.370586ms] [namespace=] [changefeed=] [2024/04/26 19:15:34.770 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348896261406731,\"CommitTs\":449348896261406736,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1\",\"tbl-id\":112,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:34.770 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348896261406731,\"CommitTs\":449348896261406736,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1\",\"tbl-id\":112,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:34.915 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="RENAME TABLE consistent_replicate_ddl.usertable_bak TO consistent_replicate_ddl.usertable1"] [duration=145.211362ms] [namespace=] [changefeed=] [2024/04/26 19:15:34.915 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348896405585928,\"CommitTs\":449348896405585931,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1_1\",\"tbl-id\":112,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:35.116 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348896405585928,\"CommitTs\":449348896405585931,\"Query\":\"RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable1_1\",\"tbl-id\":112,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:35.267 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="RENAME TABLE consistent_replicate_ddl.usertable1 TO consistent_replicate_ddl.usertable1_1"] [duration=151.668778ms] [namespace=] [changefeed=] [2024/04/26 19:15:35.267 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348896536920088,\"CommitTs\":449348896536920092,\"Query\":\"CREATE DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl1\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":1,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:35.468 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348896536920088,\"CommitTs\":449348896536920092,\"Query\":\"CREATE DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl1\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":1,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:35.553 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE DATABASE consistent_replicate_ddl1"] [duration=85.066743ms] [namespace=] [changefeed=] [2024/04/26 19:15:35.553 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348896680837144,\"CommitTs\":449348896680837150,\"Query\":\"ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":26,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:35.553 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348896680837144,\"CommitTs\":449348896680837150,\"Query\":\"ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":26,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:35.634 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER DATABASE consistent_replicate_ddl CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [duration=81.33924ms] [namespace=] [changefeed=] [2024/04/26 19:15:35.634 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348896825016334,\"CommitTs\":449348896825016339,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:35.634 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348896825016334,\"CommitTs\":449348896825016339,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:35.714 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 CHARACTER SET utf8mb4 COLLATE utf8mb4_bin"] [duration=79.375719ms] [namespace=] [changefeed=] [2024/04/26 19:15:35.714 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348897021624326,\"CommitTs\":449348897021624329,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":6,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:35.914 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348897021624326,\"CommitTs\":449348897021624329,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":6,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:36.190 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 DROP COLUMN FIELD0"] [duration=275.800323ms] [namespace=] [changefeed=] [2024/04/26 19:15:36.190 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348897152958473,\"CommitTs\":449348897152958481,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":5,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:36.190 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348897152958473,\"CommitTs\":449348897152958481,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":5,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:36.466 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable2 ADD COLUMN dummy varchar(30)"] [duration=275.61492ms] [namespace=] [changefeed=] [2024/04/26 19:15:36.466 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348897283768325,\"CommitTs\":449348897283768329,\"Query\":\"DROP DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl1\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":2,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:36.466 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348897283768325,\"CommitTs\":449348897283768329,\"Query\":\"DROP DATABASE consistent_replicate_ddl1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl1\",\"tbl-name\":\"\",\"tbl-id\":0,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":2,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:36.673 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="DROP DATABASE consistent_replicate_ddl1"] [duration=206.737989ms] [namespace=] [changefeed=] [2024/04/26 19:15:36.673 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348897388625935,\"CommitTs\":449348897388625938,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:36.673 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348897388625935,\"CommitTs\":449348897388625938,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":22,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:36.755 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable3 CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci"] [duration=82.224434ms] [namespace=] [changefeed=] [2024/04/26 19:15:36.755 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348897545912326,\"CommitTs\":449348897545912329,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3_1\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:36.755 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348897545912326,\"CommitTs\":449348897545912329,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3_1\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:36.903 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="RENAME TABLE `consistent_replicate_ddl`.`usertable3` TO `consistent_replicate_ddl`.`usertable3_1`"] [duration=147.146252ms] [namespace=] [changefeed=] [2024/04/26 19:15:36.903 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348897545912326,\"CommitTs\":449348897545912329,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2_1\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:36.903 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348897545912326,\"CommitTs\":449348897545912329,\"Query\":\"RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable2_1\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":14,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:37.047 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="RENAME TABLE `consistent_replicate_ddl`.`usertable2` TO `consistent_replicate_ddl`.`usertable2_1`"] [duration=144.60632ms] [namespace=] [changefeed=] [2024/04/26 19:15:37.047 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348897663877141,\"CommitTs\":449348897663877146,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3_1\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":12,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:37.048 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348897663877141,\"CommitTs\":449348897663877146,\"Query\":\"ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"usertable3_1\",\"tbl-id\":110,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":12,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:37.130 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE consistent_replicate_ddl.usertable3_1 MODIFY COLUMN FIELD1 varchar(100)"] [duration=82.706539ms] [namespace=] [changefeed=] [2024/04/26 19:15:37.130 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348897808056344,\"CommitTs\":449348897808056349,\"Query\":\"CREATE table consistent_replicate_ddl.check1(id int primary key)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"check1\",\"tbl-id\":127,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:15:37.332 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348897808056344,\"CommitTs\":449348897808056349,\"Query\":\"CREATE table consistent_replicate_ddl.check1(id int primary key)\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_ddl\",\"tbl-name\":\"check1\",\"tbl-id\":127,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:15:37.414 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE table consistent_replicate_ddl.check1(id int primary key)"] [duration=82.755412ms] [namespace=] [changefeed=] [2024/04/26 19:15:37.414 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}] [checkpointTs=449348897808056349] [2024/04/26 19:15:37.414 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}] [checkpointTs=449348897808056349] [2024/04/26 19:15:37.515 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}] [checkpointTs=449348897808056349] [2024/04/26 19:15:37.515 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}] [checkpointTs=449348897808056349] [2024/04/26 19:15:37.615 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=449348897808056349] [2024/04/26 19:15:37.615 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=449348897808056349] [2024/04/26 19:15:37.716 +08:00] [INFO] [redo.go:219] ["apply redo log finishes"] [appliedLogCount=101] [appliedDDLCount=14] [currentCheckpoint=449348897808056349] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/04/26 19:15:37.716 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] Apply redo log successfully table consistent_replicate_ddl.check1 exists check diff successfully [Fri Apr 26 19:15:37 CST 2024] <<<<<< run test case consistent_replicate_ddl success! >>>>>> count(*) 50 1 run task successfully table owner_resign.t1 not exists for 1-th check, retry later "is-owner": true, run task successfully test_kill_owner: pass [2024/04/26 19:15:46.777 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:46.782 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:15:46.892 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:46.973 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:46.984 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:46.988 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:46.988 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:46.991 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:46.995 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:47.002 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/spf13/pflag v1.0.5 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/stretchr/testify v1.9.0 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading github.com/tidwall/btree v1.7.0 check diff successfully wait process cdc.test exit for 1-th time... [2024/04/26 19:15:47.093 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:15:47.097 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_gbk/run.sh using Sink-Type: mysql... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 check diff failed 2-th time, retry later wait process cdc.test exit for 1-th time... check diff failed 4-th time, retry later [2024/04/26 19:15:47.285 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:47.469 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:47.475 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:47.495 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-initialize-error normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-initialize-error + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-initialize-error -s pass test: get status pass test: health pass test: changefeed apis [2024/04/26 19:15:40.830 +08:00] [INFO] [cases.go:407] ["check changefeed failed"] [time=0] [info=null] pass test: delete changefeed apis pass test: capture apis pass test: processor apis pass test: owner apis pass test: set log level + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit ++ date + echo '[Fri Apr 26 19:15:45 CST 2024] <<<<<< run test case api_v2 success! >>>>>>' [Fri Apr 26 19:15:45 CST 2024] <<<<<< run test case api_v2 success! >>>>>> + stop_tidb_cluster wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... [2024/04/26 19:15:47.571 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:47.579 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:47.687 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:47.773 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] check diff failed 1-th time, retry later + info='{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348932320362500, "checkpoint_time": "2024-04-26 19:15:43.434", "error": null }' + echo '{ "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348932320362500, "checkpoint_time": "2024-04-26 19:15:43.434", "error": null }' { "upstream_id": 7362132573846167428, "namespace": "default", "id": "changefeed-initialize-error", "state": "normal", "checkpoint_tso": 449348932320362500, "checkpoint_time": "2024-04-26 19:15:43.434", "error": null } ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449348932320362500, '"checkpoint_time":' '"2024-04-26' '19:15:43.434",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362132573846167428, '"namespace":' '"default",' '"id":' '"changefeed-initialize-error",' '"state":' '"normal",' '"checkpoint_tso":' 449348932320362500, '"checkpoint_time":' '"2024-04-26' '19:15:43.434",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_error.cli.6397.out cli changefeed remove -c changefeed-initialize-error table ddl_manager.finish_mark not exists for 40-th check, retry later You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend [2024/04/26 19:15:47.870 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:47.870 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:47.876 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:47.877 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:47.882 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:47.884 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] start tidb cluster in /tmp/tidb_cdc_test/server_config_compatibility Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_hang_up_owner [Fri Apr 26 19:15:48 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.28892891.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_owner.server1 --cluster-id default + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:15:48.184 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:15:48.272 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] wait process cdc.test exit for 3-th time... Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide [2024/04/26 19:15:48.488 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] Changefeed remove successfully. ID: changefeed-initialize-error CheckpointTs: 449348932320362500 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1 PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... table owner_resign.t1 exists [2024/04/26 19:15:48.572 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:48.591 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:48.682 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:15:48.694 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:48.779 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:15:48 CST 2024] <<<<<< run test case tiflash success! >>>>>> go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.7 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading cloud.google.com/go v0.112.0 go: downloading google.golang.org/api v0.162.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/google/btree v1.1.2 go: downloading golang.org/x/tools v0.20.0 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading golang.org/x/time v0.5.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.22.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/pkg/errors v0.9.1 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible [2024/04/26 19:15:48.875 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:48.894 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:48.898 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:48.899 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:15:48.979 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:48.991 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:48.996 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:15:48.997 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading github.com/kr/pretty v0.3.1 go: downloading cloud.google.com/go/iam v1.1.6 go: downloading cloud.google.com/go/compute v1.24.0 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/robfig/cron/v3 v3.0.1 check diff failed 3-th time, retry later [2024/04/26 19:15:49.096 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:15:49.096 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] check diff failed 1-th time, retry later check diff failed 5-th time, retry later * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17C9D049F0B535F5 < X-Xss-Protection: 1; mode=block < Date: Fri, 26 Apr 2024 11:15:49 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/robfig/cron v1.2.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/apache/thrift v0.16.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/dustin/go-humanize v1.0.1 [2024/04/26 19:15:49.380 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:49.487 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:49.571 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/rivo/uniseg v0.4.6 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da table ddl_manager.finish_mark not exists for 41-th check, retry later [2024/04/26 19:15:49.682 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:49.786 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading github.com/go-logr/stdr v1.2.2 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:15:49.973 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:49.983 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:50.074 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] + set +x check diff failed 2-th time, retry later [2024/04/26 19:15:50.097 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:50.169 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:15:50.197 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:50.280 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:50.284 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:50.287 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:50.296 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] go: downloading github.com/DataDog/zstd v1.5.5 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 [2024/04/26 19:15:50.370 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:50.477 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:50.488 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:50.492 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:50.590 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] wait process cdc.test exit for 1-th time... [2024/04/26 19:15:50.686 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:15:50.977 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:50.981 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] check diff failed 4-th time, retry later wait process cdc.test exit for 2-th time... [2024/04/26 19:15:51.089 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:51.275 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:51.282 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:51 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ea41b0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348934252101634 MinTableBarrierTs:449348934252101635 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348934252101635, checkpointTs: 449348934252101635, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348934252101635, checkpointTs: 449348934252101635, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348934252101634, checkpointTs: 449348934252101634, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106ccb31-276e-47d8-9c16-e92902e286b2 {"id":"106ccb31-276e-47d8-9c16-e92902e286b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29a7 106ccb31-276e-47d8-9c16-e92902e286b2 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348934252101634,"min-table-barrier-ts":449348934252101635,"admin-job-type":0} /tidb/cdc/default/default/task/position/106ccb31-276e-47d8-9c16-e92902e286b2/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ea41b0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348934252101634 MinTableBarrierTs:449348934252101635 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348934252101635, checkpointTs: 449348934252101635, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348934252101635, checkpointTs: 449348934252101635, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348934252101634, checkpointTs: 449348934252101634, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106ccb31-276e-47d8-9c16-e92902e286b2 {"id":"106ccb31-276e-47d8-9c16-e92902e286b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29a7 106ccb31-276e-47d8-9c16-e92902e286b2 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348934252101634,"min-table-barrier-ts":449348934252101635,"admin-job-type":0} /tidb/cdc/default/default/task/position/106ccb31-276e-47d8-9c16-e92902e286b2/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ea41b0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348934252101634 MinTableBarrierTs:449348934252101635 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348934252101635, checkpointTs: 449348934252101635, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348934252101635, checkpointTs: 449348934252101635, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348934252101634, checkpointTs: 449348934252101634, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106ccb31-276e-47d8-9c16-e92902e286b2 {"id":"106ccb31-276e-47d8-9c16-e92902e286b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29a7 106ccb31-276e-47d8-9c16-e92902e286b2 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348934252101634,"min-table-barrier-ts":449348934252101635,"admin-job-type":0} /tidb/cdc/default/default/task/position/106ccb31-276e-47d8-9c16-e92902e286b2/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2024/04/26 19:15:51.397 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:51.480 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:51.486 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:51.489 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:51.499 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:15:51.571 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] table ddl_manager.finish_mark not exists for 42-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:51 CST 2024] <<<<<< run test case changefeed_error success! >>>>>> check diff failed 6-th time, retry later [2024/04/26 19:15:51.679 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:51.684 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:51.686 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:51.779 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:51.792 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] check diff successfully [Fri Apr 26 19:15:51 CST 2024] <<<<<< START cdc server in owner_resign case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerDontUpdateCheckpoint=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_resign.25512553.out server --log-file /tmp/tidb_cdc_test/owner_resign/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_resign/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2024/04/26 19:15:51.897 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:51.899 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:51.987 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:52.074 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:52.080 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] check diff failed 3-th time, retry later Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_gbk Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:15:52.476 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:52.490 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:52.574 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:52.577 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:52.590 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:52.595 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:15:52.696 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] "is-owner": true, run task successfully [2024/04/26 19:15:52.780 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:52.782 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:52.870 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:52.890 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] check diff failed 5-th time, retry later [2024/04/26 19:15:52.992 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:52.994 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:53.082 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:53.087 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:53.096 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:15:51.162 +08:00] [INFO] [case.go:445] ["open db success"] [dsn=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:15:51.163 +08:00] [INFO] [case.go:445] ["open db success"] [dsn=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:15:51.164 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq0] [2024/04/26 19:15:51.254 +08:00] [INFO] [case.go:365] ["insert 100 takes 7.844229ms"] [query="INSERT IGNORE INTO accounts_seq0 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/04/26 19:15:51.255 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq1] [2024/04/26 19:15:51.345 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.285319ms"] [query="INSERT IGNORE INTO accounts_seq1 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/04/26 19:15:51.346 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq2] [2024/04/26 19:15:51.438 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.708922ms"] [query="INSERT IGNORE INTO accounts_seq2 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/04/26 19:15:51.438 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq3] [2024/04/26 19:15:51.529 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.527181ms"] [query="INSERT IGNORE INTO accounts_seq3 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/04/26 19:15:51.529 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts_seq4] [2024/04/26 19:15:51.622 +08:00] [INFO] [case.go:365] ["insert 100 takes 9.584654ms"] [query="INSERT IGNORE INTO accounts_seq4 (id, counter, sequence, startts) VALUES (0, 0, 0, 0),(1, 0, 0, 0),(2, 0, 0, 0),(3, 0, 0, 0),(4, 0, 0, 0),(5, 0, 0, 0),(6, 0, 0, 0),(7, 0, 0, 0),(8, 0, 0, 0),(9, 0, 0, 0),(10, 0, 0, 0),(11, 0, 0, 0),(12, 0, 0, 0),(13, 0, 0, 0),(14, 0, 0, 0),(15, 0, 0, 0),(16, 0, 0, 0),(17, 0, 0, 0),(18, 0, 0, 0),(19, 0, 0, 0),(20, 0, 0, 0),(21, 0, 0, 0),(22, 0, 0, 0),(23, 0, 0, 0),(24, 0, 0, 0),(25, 0, 0, 0),(26, 0, 0, 0),(27, 0, 0, 0),(28, 0, 0, 0),(29, 0, 0, 0),(30, 0, 0, 0),(31, 0, 0, 0),(32, 0, 0, 0),(33, 0, 0, 0),(34, 0, 0, 0),(35, 0, 0, 0),(36, 0, 0, 0),(37, 0, 0, 0),(38, 0, 0, 0),(39, 0, 0, 0),(40, 0, 0, 0),(41, 0, 0, 0),(42, 0, 0, 0),(43, 0, 0, 0),(44, 0, 0, 0),(45, 0, 0, 0),(46, 0, 0, 0),(47, 0, 0, 0),(48, 0, 0, 0),(49, 0, 0, 0),(50, 0, 0, 0),(51, 0, 0, 0),(52, 0, 0, 0),(53, 0, 0, 0),(54, 0, 0, 0),(55, 0, 0, 0),(56, 0, 0, 0),(57, 0, 0, 0),(58, 0, 0, 0),(59, 0, 0, 0),(60, 0, 0, 0),(61, 0, 0, 0),(62, 0, 0, 0),(63, 0, 0, 0),(64, 0, 0, 0),(65, 0, 0, 0),(66, 0, 0, 0),(67, 0, 0, 0),(68, 0, 0, 0),(69, 0, 0, 0),(70, 0, 0, 0),(71, 0, 0, 0),(72, 0, 0, 0),(73, 0, 0, 0),(74, 0, 0, 0),(75, 0, 0, 0),(76, 0, 0, 0),(77, 0, 0, 0),(78, 0, 0, 0),(79, 0, 0, 0),(80, 0, 0, 0),(81, 0, 0, 0),(82, 0, 0, 0),(83, 0, 0, 0),(84, 0, 0, 0),(85, 0, 0, 0),(86, 0, 0, 0),(87, 0, 0, 0),(88, 0, 0, 0),(89, 0, 0, 0),(90, 0, 0, 0),(91, 0, 0, 0),(92, 0, 0, 0),(93, 0, 0, 0),(94, 0, 0, 0),(95, 0, 0, 0),(96, 0, 0, 0),(97, 0, 0, 0),(98, 0, 0, 0),(99, 0, 0, 0)"] [2024/04/26 19:15:51.622 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts0] [2024/04/26 19:15:51.713 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.09315ms"] [query="INSERT IGNORE INTO accounts0 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/04/26 19:15:51.714 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts1] [2024/04/26 19:15:51.808 +08:00] [INFO] [case.go:365] ["insert 100 takes 10.690376ms"] [query="INSERT IGNORE INTO accounts1 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/04/26 19:15:51.809 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts2] [2024/04/26 19:15:51.902 +08:00] [INFO] [case.go:365] ["insert 100 takes 9.23974ms"] [query="INSERT IGNORE INTO accounts2 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/04/26 19:15:51.902 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts3] [2024/04/26 19:15:51.992 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.566955ms"] [query="INSERT IGNORE INTO accounts3 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/04/26 19:15:51.993 +08:00] [INFO] [case.go:383] ["drop tables"] [table=accounts4] [2024/04/26 19:15:52.083 +08:00] [INFO] [case.go:365] ["insert 100 takes 8.660739ms"] [query="INSERT IGNORE INTO accounts4 (id, balance, startts) VALUES (0, 1000, 0),(1, 1000, 0),(2, 1000, 0),(3, 1000, 0),(4, 1000, 0),(5, 1000, 0),(6, 1000, 0),(7, 1000, 0),(8, 1000, 0),(9, 1000, 0),(10, 1000, 0),(11, 1000, 0),(12, 1000, 0),(13, 1000, 0),(14, 1000, 0),(15, 1000, 0),(16, 1000, 0),(17, 1000, 0),(18, 1000, 0),(19, 1000, 0),(20, 1000, 0),(21, 1000, 0),(22, 1000, 0),(23, 1000, 0),(24, 1000, 0),(25, 1000, 0),(26, 1000, 0),(27, 1000, 0),(28, 1000, 0),(29, 1000, 0),(30, 1000, 0),(31, 1000, 0),(32, 1000, 0),(33, 1000, 0),(34, 1000, 0),(35, 1000, 0),(36, 1000, 0),(37, 1000, 0),(38, 1000, 0),(39, 1000, 0),(40, 1000, 0),(41, 1000, 0),(42, 1000, 0),(43, 1000, 0),(44, 1000, 0),(45, 1000, 0),(46, 1000, 0),(47, 1000, 0),(48, 1000, 0),(49, 1000, 0),(50, 1000, 0),(51, 1000, 0),(52, 1000, 0),(53, 1000, 0),(54, 1000, 0),(55, 1000, 0),(56, 1000, 0),(57, 1000, 0),(58, 1000, 0),(59, 1000, 0),(60, 1000, 0),(61, 1000, 0),(62, 1000, 0),(63, 1000, 0),(64, 1000, 0),(65, 1000, 0),(66, 1000, 0),(67, 1000, 0),(68, 1000, 0),(69, 1000, 0),(70, 1000, 0),(71, 1000, 0),(72, 1000, 0),(73, 1000, 0),(74, 1000, 0),(75, 1000, 0),(76, 1000, 0),(77, 1000, 0),(78, 1000, 0),(79, 1000, 0),(80, 1000, 0),(81, 1000, 0),(82, 1000, 0),(83, 1000, 0),(84, 1000, 0),(85, 1000, 0),(86, 1000, 0),(87, 1000, 0),(88, 1000, 0),(89, 1000, 0),(90, 1000, 0),(91, 1000, 0),(92, 1000, 0),(93, 1000, 0),(94, 1000, 0),(95, 1000, 0),(96, 1000, 0),(97, 1000, 0),(98, 1000, 0),(99, 1000, 0)"] [2024/04/26 19:15:53.385 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] Cache restored successfully (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc) 3365111296 bytes in 7.10 secs (474075779 bytes/sec) [Pipeline] { [Pipeline] sh [2024/04/26 19:15:53.573 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:53.582 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:53.583 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:53.682 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] check diff successfully go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading github.com/felixge/httpsnoop v1.0.4 [2024/04/26 19:15:53.774 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:15:53.976 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] check diff failed 1-th time, retry later go: downloading github.com/jmespath/go-jmespath v0.4.0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // node [Pipeline] } table ddl_manager.finish_mark not exists for 43-th check, retry later [2024/04/26 19:15:54.164 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2024/04/26 19:15:54.076 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:54.096 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] owner pid: 2894 owner id 106ccb31-276e-47d8-9c16-e92902e286b2 [Fri Apr 26 19:15:54 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.30003002.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_owner.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_owner.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // node [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:54 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/8291dabe-e8f7-4a20-a12b-aca8243b981d {UpstreamID:7362132787447678527 Namespace:default ID:8291dabe-e8f7-4a20-a12b-aca8243b981d SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:44.144463822 +0800 CST StartTs:449348932454055938 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031321b0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348932506222593} {CheckpointTs:449348934957269003 MinTableBarrierTs:449348934957269003 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f87aa33-dd9a-4a87-b81c-a6d30183bba2 {"id":"4f87aa33-dd9a-4a87-b81c-a6d30183bba2","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/918cfa20-1c58-478b-a2b6-472613736bea {"id":"918cfa20-1c58-478b-a2b6-472613736bea","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b0a 4f87aa33-dd9a-4a87-b81c-a6d30183bba2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b81 918cfa20-1c58-478b-a2b6-472613736bea /tidb/cdc/default/default/changefeed/info/8291dabe-e8f7-4a20-a12b-aca8243b981d {"upstream-id":7362132787447678527,"namespace":"default","changefeed-id":"8291dabe-e8f7-4a20-a12b-aca8243b981d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:44.144463822+08:00","start-ts":449348932454055938,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348932506222593} /tidb/cdc/default/default/changefeed/status/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":449348934957269003,"min-table-barrier-ts":449348934957269003,"admin-job-type":0} /tidb/cdc/default/default/task/position/4f87aa33-dd9a-4a87-b81c-a6d30183bba2/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/918cfa20-1c58-478b-a2b6-472613736bea/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132787447678527 {"id":7362132787447678527,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/8291dabe-e8f7-4a20-a12b-aca8243b981d {UpstreamID:7362132787447678527 Namespace:default ID:8291dabe-e8f7-4a20-a12b-aca8243b981d SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:44.144463822 +0800 CST StartTs:449348932454055938 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031321b0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348932506222593} {CheckpointTs:449348934957269003 MinTableBarrierTs:449348934957269003 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f87aa33-dd9a-4a87-b81c-a6d30183bba2 {"id":"4f87aa33-dd9a-4a87-b81c-a6d30183bba2","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/918cfa20-1c58-478b-a2b6-472613736bea {"id":"918cfa20-1c58-478b-a2b6-472613736bea","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b0a 4f87aa33-dd9a-4a87-b81c-a6d30183bba2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b81 918cfa20-1c58-478b-a2b6-472613736bea /tidb/cdc/default/default/changefeed/info/8291dabe-e8f7-4a20-a12b-aca8243b981d {"upstream-id":7362132787447678527,"namespace":"default","changefeed-id":"8291dabe-e8f7-4a20-a12b-aca8243b981d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:44.144463822+08:00","start-ts":449348932454055938,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348932506222593} /tidb/cdc/default/default/changefeed/status/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":449348934957269003,"min-table-barrier-ts":449348934957269003,"admin-job-type":0} /tidb/cdc/default/default/task/position/4f87aa33-dd9a-4a87-b81c-a6d30183bba2/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/918cfa20-1c58-478b-a2b6-472613736bea/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132787447678527 {"id":7362132787447678527,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/8291dabe-e8f7-4a20-a12b-aca8243b981d {UpstreamID:7362132787447678527 Namespace:default ID:8291dabe-e8f7-4a20-a12b-aca8243b981d SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:44.144463822 +0800 CST StartTs:449348932454055938 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031321b0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348932506222593} {CheckpointTs:449348934957269003 MinTableBarrierTs:449348934957269003 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4f87aa33-dd9a-4a87-b81c-a6d30183bba2 {"id":"4f87aa33-dd9a-4a87-b81c-a6d30183bba2","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/918cfa20-1c58-478b-a2b6-472613736bea {"id":"918cfa20-1c58-478b-a2b6-472613736bea","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b0a 4f87aa33-dd9a-4a87-b81c-a6d30183bba2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dd23b81 918cfa20-1c58-478b-a2b6-472613736bea /tidb/cdc/default/default/changefeed/info/8291dabe-e8f7-4a20-a12b-aca8243b981d {"upstream-id":7362132787447678527,"namespace":"default","changefeed-id":"8291dabe-e8f7-4a20-a12b-aca8243b981d","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:44.144463822+08:00","start-ts":449348932454055938,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348932506222593} /tidb/cdc/default/default/changefeed/status/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":449348934957269003,"min-table-barrier-ts":449348934957269003,"admin-job-type":0} /tidb/cdc/default/default/task/position/4f87aa33-dd9a-4a87-b81c-a6d30183bba2/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/918cfa20-1c58-478b-a2b6-472613736bea/8291dabe-e8f7-4a20-a12b-aca8243b981d {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132787447678527 {"id":7362132787447678527,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^2$' [2024/04/26 19:15:54.286 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:54.299 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:54.375 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:54.383 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:15:54.491 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:54.495 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] check diff failed 4-th time, retry later 2 run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [2024/04/26 19:15:54.506 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:54.574 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:54.583 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:15:54.672 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:54.699 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd [2024/04/26 19:15:54.787 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:55.082 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:55.088 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:55.174 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:15:55.493 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:55.497 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] check diff failed 6-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_adapter_compatibility/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:15:55 CST 2024] <<<<<< run test case canal_json_adapter_compatibility success! >>>>>> [2024/04/26 19:15:55.576 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:55.587 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:55.592 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:15:55.687 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:15:55.707 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] + rm -rf /tmp/tidb_cdc_test + mkdir -p /tmp/tidb_cdc_test + chmod +x ./tests/integration_tests/run_group.sh + ./tests/integration_tests/run_group.sh mysql G19 Run cases: changefeed_fast_fail batch_update_to_no_batch changefeed_resume_with_checkpoint_ts PROW_JOB_ID=d18e9f95-32a3-42a7-8fe2-3b7a27f83b68 JENKINS_NODE_COOKIE=c2ab85b2-da2b-4a1d-b348-1df9f1770702 BUILD_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/ GOLANG_VERSION=1.21.6 HOSTNAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g HUDSON_SERVER_COOKIE=83ef27fe9acccc92 KUBERNETES_PORT_443_TCP_PORT=443 KUBERNETES_PORT=tcp://10.233.0.1:443 TERM=xterm STAGE_NAME=Test BUILD_TAG=jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303 KUBERNETES_SERVICE_PORT=443 GIT_PREVIOUS_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 JOB_SPEC={"type":"presubmit","job":"pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test","buildid":"1783812214621212672","prowjobid":"d18e9f95-32a3-42a7-8fe2-3b7a27f83b68","refs":{"org":"pingcap","repo":"tiflow","repo_link":"https://github.com/pingcap/tiflow","base_ref":"release-7.5","base_sha":"823a3899cc17bf14ad7875ab0cc69092186abc4a","base_link":"https://github.com/pingcap/tiflow/commit/823a3899cc17bf14ad7875ab0cc69092186abc4a","pulls":[{"number":10970,"author":"3AceShowHand","sha":"aa323aa99660421afc797ea115dbb30242e573cf","title":"*: release-7.5 bump tidb dependencies","link":"https://github.com/pingcap/tiflow/pull/10970","commit_link":"https://github.com/pingcap/tiflow/pull/10970/commits/aa323aa99660421afc797ea115dbb30242e573cf","author_link":"https://github.com/3AceShowHand"}]}} KUBERNETES_SERVICE_HOST=10.233.0.1 WORKSPACE=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test JOB_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/ RUN_CHANGES_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=changes RUN_ARTIFACTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=artifacts FILE_SERVER_URL=http://fileserver.pingcap.net JENKINS_HOME=/var/jenkins_home GIT_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../bin:/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/../../scripts/bin RUN_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect GOPROXY=http://goproxy.apps.svc,https://proxy.golang.org,direct POD_CONTAINER=golang PWD=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow HUDSON_URL=https://do.pingcap.net/jenkins/ TICDC_COVERALLS_TOKEN=**** JOB_NAME=pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test TZ=Asia/Shanghai BUILD_DISPLAY_NAME=#303 TEST_GROUP=G19 JENKINS_URL=https://do.pingcap.net/jenkins/ BUILD_ID=1783812214621212672 TICDC_CODECOV_TOKEN=**** GOLANG_DOWNLOAD_SHA256=3f934f40ac360b9c01f616a9aa1796d227d8b0328bf64cb045c7b8c4ee9caea4 JOB_BASE_NAME=pull_cdc_integration_mysql_test GIT_PREVIOUS_SUCCESSFUL_COMMIT=cb9bc9e8822a01a5d59a2f670fb429e588065145 RUN_TESTS_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/303/display/redirect?page=tests SHLVL=5 HOME=/home/jenkins POD_LABEL=ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-fw3w4 GOROOT=/usr/local/go GIT_BRANCH=origin/main KUBERNETES_PORT_443_TCP_PROTO=tcp CI=true KUBERNETES_SERVICE_PORT_HTTPS=443 WORKSPACE_TMP=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test@tmp EXECUTOR_NUMBER=0 JENKINS_SERVER_COOKIE=durable-03d44d992f53cff0b3bb7ab1ee8dd6585d62f1b48625706d61992f6495e84f7f NODE_LABELS=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g ap_tiflow_release-7_5_pull_cdc_integration_mysql_test_303-fw3w4 GIT_URL=https://github.com/PingCAP-QE/ci.git HUDSON_HOME=/var/jenkins_home CLASSPATH= NODE_NAME=ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g GOPATH=/go JOB_DISPLAY_URL=https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/job/release-7.5/job/pull_cdc_integration_mysql_test/display/redirect BUILD_NUMBER=303 KUBERNETES_PORT_443_TCP_ADDR=10.233.0.1 KUBERNETES_PORT_443_TCP=tcp://10.233.0.1:443 GOLANG_DOWNLOAD_URL=https://dl.google.com/go/go1.21.6.linux-amd64.tar.gz _=/usr/bin/env find: '/tmp/tidb_cdc_test/*/*': No such file or directory =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_fast_fail/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/04/26 19:15:55.880 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:55.887 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/04/26 19:15:55.893 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:15:56.171 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:56.176 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:56.286 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] check diff successfully [2024/04/26 19:15:56.388 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:15:56.471 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsfe5c3401_da3f_4c27_bb37_de253650d1fc"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later [2024/04/26 19:15:56.177 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] wait process cdc.test exit for 1-th time... [2024/04/26 19:15:56.584 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:56.678 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:15:56.875 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:15:56.982 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:56.990 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] wait process cdc.test exit for 2-th time... [2024/04/26 19:15:57.080 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:15:57.091 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:57.095 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:57.112 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/04/26 19:15:57.176 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLse13bb99d_a084_41eb_86e8_7fcd1bb4b53c"] [2024/04/26 19:15:57.287 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsd06d6fb8_53aa_4a84_ae8b_5032b5a21f2f"] [2024/04/26 19:15:57.293 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] check diff failed 7-th time, retry later table ddl_manager.finish_mark not exists for 44-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:15:57 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0032a30e0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348935562821638 MinTableBarrierTs:449348935824965638 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106ccb31-276e-47d8-9c16-e92902e286b2 {"id":"106ccb31-276e-47d8-9c16-e92902e286b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/72f938fd-cf3e-4939-8bbd-e22f7245584d {"id":"72f938fd-cf3e-4939-8bbd-e22f7245584d","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29a7 106ccb31-276e-47d8-9c16-e92902e286b2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29de 72f938fd-cf3e-4939-8bbd-e22f7245584d /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348935824965637,"min-table-barrier-ts":449348935824965638,"admin-job-type":0} /tidb/cdc/default/default/task/position/106ccb31-276e-47d8-9c16-e92902e286b2/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/72f938fd-cf3e-4939-8bbd-e22f7245584d/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0032a30e0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348935562821638 MinTableBarrierTs:449348935824965638 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106ccb31-276e-47d8-9c16-e92902e286b2 {"id":"106ccb31-276e-47d8-9c16-e92902e286b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/72f938fd-cf3e-4939-8bbd-e22f7245584d {"id":"72f938fd-cf3e-4939-8bbd-e22f7245584d","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29a7 106ccb31-276e-47d8-9c16-e92902e286b2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29de 72f938fd-cf3e-4939-8bbd-e22f7245584d /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348935824965637,"min-table-barrier-ts":449348935824965638,"admin-job-type":0} /tidb/cdc/default/default/task/position/106ccb31-276e-47d8-9c16-e92902e286b2/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/72f938fd-cf3e-4939-8bbd-e22f7245584d/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0032a30e0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348935562821638 MinTableBarrierTs:449348935824965638 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/106ccb31-276e-47d8-9c16-e92902e286b2 {"id":"106ccb31-276e-47d8-9c16-e92902e286b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/72f938fd-cf3e-4939-8bbd-e22f7245584d {"id":"72f938fd-cf3e-4939-8bbd-e22f7245584d","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29a7 106ccb31-276e-47d8-9c16-e92902e286b2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce29de 72f938fd-cf3e-4939-8bbd-e22f7245584d /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348935824965637,"min-table-barrier-ts":449348935824965638,"admin-job-type":0} /tidb/cdc/default/default/task/position/106ccb31-276e-47d8-9c16-e92902e286b2/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/72f938fd-cf3e-4939-8bbd-e22f7245584d/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 | grep -v "106ccb31-276e-47d8-9c16-e92902e286b2" | grep id cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:15:57 CST 2024] <<<<<< run test case new_ci_collation success! >>>>>> [2024/04/26 19:15:57.575 +08:00] [INFO] [main.go:178] ["73 insert success: 1600"] [2024/04/26 19:15:57.575 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLsb5385238_24a3_4eb4_b04a_b0b7035c0ede"] [2024/04/26 19:15:57.890 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:15:57.972 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:57.977 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:57.995 +08:00] [INFO] [main.go:178] ["72 insert success: 1700"] [2024/04/26 19:15:58.085 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] Starting Upstream TiDB... [2024/04/26 19:15:58.183 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs4cd13f6e_9339_44f6_a5af_b662b058e92f"] [2024/04/26 19:15:58.277 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:58.281 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:15:58.287 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:58.288 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:58.392 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6879fd4c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:10056, start at 2024-04-26 19:15:57.401031528 +0800 CST m=+5.458342741 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:57.408 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:57.395 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:57.395 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully [2024/04/26 19:15:58.189 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2024/04/26 19:15:58.474 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [Pipeline] // podTemplate "id": "72f938fd-cf3e-4939-8bbd-e22f7245584d", "cluster-id": "default" "cluster-id": "default" run task successfully [2024/04/26 19:15:58.685 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs85300dde_2f5e_4331_b5ad_f78f178f2517"] [2024/04/26 19:15:58.769 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:15:58.777 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:15:58.878 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:15:58 CST 2024] <<<<<< run test case canal_json_basic success! >>>>>> [Pipeline] // podTemplate [2024/04/26 19:15:59.079 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] check diff failed 8-th time, retry later table ddl_manager.finish_mark not exists for 45-th check, retry later [2024/04/26 19:15:59.280 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:59.472 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] [2024/04/26 19:15:59.478 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:59.483 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:59.485 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:59.491 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:59.592 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:15:59.684 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:15:59.893 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:15:59.898 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] start tidb cluster in /tmp/tidb_cdc_test/changefeed_fast_fail Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:15:59.969 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:15:59.994 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:00.009 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:00.095 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:16:00.197 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:00.200 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] capture_id: 72f938fd-cf3e-4939-8bbd-e22f7245584d ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '72f938fd-cf3e-4939-8bbd-e22f7245584d' 72f938fd-cf3e-4939-8bbd-e22f7245584d run task successfully test_hang_up_owner: pass ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:00.269 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:00.282 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:00.375 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:00.201 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] wait process cdc.test exit for 1-th time... [Pipeline] } [2024/04/26 19:16:00.485 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [Pipeline] // withEnv [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6879fd4c0006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:10056, start at 2024-04-26 19:15:57.401031528 +0800 CST m=+5.458342741 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:57.408 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:57.395 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:57.395 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6879fc740005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-46k73, pid:10141, start at 2024-04-26 19:15:57.345303952 +0800 CST m=+5.315466369 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:17:57.353 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:15:57.341 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:05:57.341 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/error.log [2024/04/26 19:16:00.881 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/server_config_compatibility/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/04/26 19:16:00.896 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:00.901 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] wait process cdc.test exit for 2-th time... [2024/04/26 19:16:00.992 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:01.001 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:01.088 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:01.103 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:01.180 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:01.183 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:01.187 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:01.193 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] table ddl_manager.finish_mark not exists for 46-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_topics/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:01 CST 2024] <<<<<< run test case multi_topics success! >>>>>> [2024/04/26 19:16:01.293 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:01.390 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs82cb4e97_1256_4dd2_934d_c2236b4ab1c8"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_expire_owner [Fri Apr 26 19:16:01 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.31323134.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_expire_owner.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_expire_owner.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_sequence/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] // withEnv [2024/04/26 19:16:01.587 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [Pipeline] } check diff failed 9-th time, retry later [Pipeline] // stage [2024/04/26 19:16:01.777 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:02.080 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:02.175 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:02.189 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [Pipeline] } [2024/04/26 19:16:02.214 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [Pipeline] // stage [Pipeline] } [2024/04/26 19:16:02.286 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:02.377 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:02.481 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:02.483 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:02.490 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:02.496 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] check diff failed 1-th time, retry later [2024/04/26 19:16:02.599 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:02.676 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:16:02.772 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:16:02.777 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:02.790 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs454d72cc_995e_4a4b_9f15_f4fba4645e3d"] + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.cli.11523.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later [2024/04/26 19:16:02.969 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] table ddl_manager.finish_mark not exists for 47-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... [2024/04/26 19:16:03.283 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:03.477 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:03.488 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] check diff failed 10-th time, retry later [2024/04/26 19:16:03.595 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:03.671 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:03.777 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:03.881 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:03.895 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:03.898 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:16:03.969 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:03.987 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:03.990 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:03.995 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:16:04.072 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:04.073 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] start tidb cluster in /tmp/tidb_cdc_test/ddl_sequence Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:16:04.226 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] check diff successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [2024/04/26 19:16:04.085 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:16:04.195 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] check diff failed 2-th time, retry later [2024/04/26 19:16:04.381 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:04.385 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:04.488 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:04.584 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/avro_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:04 CST 2024] <<<<<< run test case avro_basic success! >>>>>> + set +x + tso='449348937422209025 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348937422209025 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:16:04 CST 2024] <<<<<< START cdc server in server_config_compatibility case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.1156911571.out server --log-file /tmp/tidb_cdc_test/server_config_compatibility/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/server_config_compatibility/cdc_data --cluster-id default --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/server_config_compatibility/conf/server.toml + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:04.597 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:04.789 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:04 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c926c0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348937659973635 MinTableBarrierTs:449348937659973636 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348937659973636, checkpointTs: 449348937659973636, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348937659973636, checkpointTs: 449348937659973636, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348937659973635, checkpointTs: 449348937659973635, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8 {"id":"c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a18 c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348937659973635,"min-table-barrier-ts":449348937659973636,"admin-job-type":0} /tidb/cdc/default/default/task/position/c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c926c0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348937659973635 MinTableBarrierTs:449348937659973636 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348937659973636, checkpointTs: 449348937659973636, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348937659973636, checkpointTs: 449348937659973636, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348937659973635, checkpointTs: 449348937659973635, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8 {"id":"c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a18 c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348937659973635,"min-table-barrier-ts":449348937659973636,"admin-job-type":0} /tidb/cdc/default/default/task/position/c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002c926c0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348937659973635 MinTableBarrierTs:449348937659973636 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348937659973636, checkpointTs: 449348937659973636, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348937659973636, checkpointTs: 449348937659973636, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348937659973635, checkpointTs: 449348937659973635, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8 {"id":"c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a18 c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348937659973635,"min-table-barrier-ts":449348937659973636,"admin-job-type":0} /tidb/cdc/default/default/task/position/c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' table ddl_manager.finish_mark not exists for 48-th check, retry later [2024/04/26 19:16:04.876 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:04.883 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:04.889 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:04.895 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:04.979 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:16:04.980 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:04.998 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:05.070 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:05.079 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:16:05.171 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:05.280 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:05.287 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 11-th time, retry later [2024/04/26 19:16:05.389 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:05.578 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:05.581 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:05.978 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:05.981 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:06.082 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:06.085 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:06.175 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:06.177 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] "is-owner": true, run task successfully build mysql_test successfully run all mysql test cases time="2024-04-26T19:16:02+08:00" level=info msg="running tests: [alter_table bigint composite_index concurrent_ddl date_formats drop mysql_replace]" time="2024-04-26T19:16:02+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000126000 {{} {} 0} {0 0} [0xc000140000] map[] 0 1 0xc00012a000 false map[0xc000140000:map[0xc000140000:true]] map[] 0 0 0 0 0 0 0 0 0x4914e0}" check diff failed 3-th time, retry later [2024/04/26 19:16:06.195 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:06.271 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:16:06.282 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:06.290 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:06.372 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687a5c30000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:4144, start at 2024-04-26 19:16:03.486435867 +0800 CST m=+5.277442235 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:03.493 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:03.468 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:03.468 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687a5c30000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:4144, start at 2024-04-26 19:16:03.486435867 +0800 CST m=+5.277442235 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:03.493 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:03.468 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:03.468 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687a5dac0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:4227, start at 2024-04-26 19:16:03.599924051 +0800 CST m=+5.326111444 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:03.608 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:03.613 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:03.613 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_gbk/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:16:06.473 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:06.489 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:06.581 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:06.670 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:16:06.690 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:06.241 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2024/04/26 19:16:06.794 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:06.889 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:06.889 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/region_merge/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/04/26 19:16:06.974 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:06.978 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:07.077 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:07.081 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:07.184 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:07 CST 2024] <<<<<< run test case canal_json_handle_key_only success! >>>>>> [2024/04/26 19:16:07.271 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:07.273 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/batch_add_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table ddl_manager.finish_mark not exists for 49-th check, retry later owner pid: 3137 owner id c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8 process status: T ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep -v 'c53bfd8b-80b4-4a93-aa1a-9f4c2d8847c8' /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a18 run task successfully process status: S cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2024/04/26 19:16:07.473 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:16:07.580 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:07.769 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:07.787 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:16:07.791 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:16:07.969 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:07 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c4f9d4b4-7a36-4e8e-be52-d65f4d512f94 {"id":"c4f9d4b4-7a36-4e8e-be52-d65f4d512f94","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e561809 c4f9d4b4-7a36-4e8e-be52-d65f4d512f94 /tidb/cdc/default/default/upstream/7362132929435315267 {"id":7362132929435315267,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c4f9d4b4-7a36-4e8e-be52-d65f4d512f94 {"id":"c4f9d4b4-7a36-4e8e-be52-d65f4d512f94","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e561809 c4f9d4b4-7a36-4e8e-be52-d65f4d512f94 /tidb/cdc/default/default/upstream/7362132929435315267 {"id":7362132929435315267,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c4f9d4b4-7a36-4e8e-be52-d65f4d512f94 {"id":"c4f9d4b4-7a36-4e8e-be52-d65f4d512f94","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e561809 c4f9d4b4-7a36-4e8e-be52-d65f4d512f94 /tidb/cdc/default/default/upstream/7362132929435315267 {"id":7362132929435315267,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.server_config_compatibility.cli.11618.out cli changefeed create --start-ts=449348937422209025 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ check diff failed 12-th time, retry later [2024/04/26 19:16:08.078 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:08.173 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:16:08.177 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:08.177 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:16:08.180 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:08.193 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] Create changefeed successfully! ID: 4c237bf6-5c95-40fb-a15c-feef2ef7b51a Info: {"upstream_id":7362132929435315267,"namespace":"default","id":"4c237bf6-5c95-40fb-a15c-feef2ef7b51a","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:16:08.012763717+08:00","start_ts":449348937422209025,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348937422209025,"checkpoint_ts":449348937422209025,"checkpoint_time":"2024-04-26 19:16:02.896"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2024/04/26 19:16:08.282 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:08.288 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:08.290 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:08.373 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:16:08.472 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:08.253 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2024/04/26 19:16:08.575 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:08.688 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] check diff failed 4-th time, retry later "is-owner": true, run task successfully test_expire_owner pass [2024/04/26 19:16:08.788 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:16:08.788 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:16:08.893 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [Fri Apr 26 19:16:08 CST 2024] <<<<<< START cdc server in consistent_replicate_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_gbk.55135515.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... [2024/04/26 19:16:09.087 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:16:09.182 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:16:09.183 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:16:09.184 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:09.185 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:16:09.185 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] table ddl_manager.finish_mark not exists for 50-th check, retry later [2024/04/26 19:16:09.287 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:09.291 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:09.295 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:09.375 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:16:09.390 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:16:09.479 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] start tidb cluster in /tmp/tidb_cdc_test/region_merge Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... + set +x [2024/04/26 19:16:09.673 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:16:09.692 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:16:09.695 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687aacb0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:1161, start at 2024-04-26 19:16:08.638825756 +0800 CST m=+5.368953829 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:08.648 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:08.620 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:08.620 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later check diff failed 13-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/batch_add_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:16:09.793 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:16:09.972 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:16:09.991 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:09.995 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:16:10.072 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:16:10.076 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:16:10.084 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:16:10.093 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:10.187 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:10.189 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:10.190 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:16:10.283 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_owner_cleanup_stale_tasks [Fri Apr 26 19:16:10 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.32963298.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/open_protocol_handle_key_only/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:10 CST 2024] <<<<<< run test case open_protocol_handle_key_only success! >>>>>> [2024/04/26 19:16:10.265 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2024/04/26 19:16:10.392 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] check diff failed 5-th time, retry later [2024/04/26 19:16:10.581 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:16:10.878 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:16:10.990 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:11.184 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:16:11.186 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:11.190 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:16:11.278 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:11.289 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:16:11.295 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] table ddl_manager.finish_mark not exists for 51-th check, retry later Verifying downstream PD is started... [2024/04/26 19:16:11.371 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:16:11.389 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:16:11.392 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:11.396 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:11.572 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:16:11.577 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:11 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f04d54a-f77c-4b7c-a30b-251c0da234e0 {"id":"2f04d54a-f77c-4b7c-a30b-251c0da234e0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e6ca3f2 2f04d54a-f77c-4b7c-a30b-251c0da234e0 /tidb/cdc/default/default/upstream/7362132960996133459 {"id":7362132960996133459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f04d54a-f77c-4b7c-a30b-251c0da234e0 {"id":"2f04d54a-f77c-4b7c-a30b-251c0da234e0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e6ca3f2 2f04d54a-f77c-4b7c-a30b-251c0da234e0 /tidb/cdc/default/default/upstream/7362132960996133459 {"id":7362132960996133459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2f04d54a-f77c-4b7c-a30b-251c0da234e0 {"id":"2f04d54a-f77c-4b7c-a30b-251c0da234e0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e6ca3f2 2f04d54a-f77c-4b7c-a30b-251c0da234e0 /tidb/cdc/default/default/upstream/7362132960996133459 {"id":7362132960996133459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687aacb0000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:1161, start at 2024-04-26 19:16:08.638825756 +0800 CST m=+5.368953829 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:08.648 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:08.620 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:08.620 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687abea00014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:1243, start at 2024-04-26 19:16:09.808599696 +0800 CST m=+6.463291474 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:09.816 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:09.819 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:09.819 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_fast_fail/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later check diff failed 14-th time, retry later [2024/04/26 19:16:11.692 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:16:12.278 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark] [2024/04/26 19:16:12.294 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark] [ts=449348939755028485] [2024/04/26 19:16:12.294 +08:00] [INFO] [case.go:498] ["all tables synced"] [endTs=449348939755028485] [2024/04/26 19:16:12.173 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:16:12.182 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:16:12.183 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:16:12.200 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:12.279 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:16:12.295 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:16:12.300 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:16:12.382 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:12.391 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:16:12.394 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:16:12.481 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:16:12.483 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:16:12.781 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs17b5c656_8591_49c1_ae1b_747aa1ed82bf"] [2024/04/26 19:16:12.794 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:16:12.802 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:16:12.876 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] [2024/04/26 19:16:12.891 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:12.988 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:16:13.000 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:16:13.001 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:16:13.100 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] check diff successfully table ddl_manager.finish_mark not exists for 52-th check, retry later [2024/04/26 19:16:13.381 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:16:13.384 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:16:13.395 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:16:13.396 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] table consistent_replicate_gbk.GBKTABLE not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:13 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0030d2990 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348940019007492 MinTableBarrierTs:449348940019007493 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348940019007493, checkpointTs: 449348940019007493, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348940019007493, checkpointTs: 449348940019007493, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348940019007492, checkpointTs: 449348940019007492, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348940019007492,"min-table-barrier-ts":449348940019007493,"admin-job-type":0} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0030d2990 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348940019007492 MinTableBarrierTs:449348940019007493 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348940019007493, checkpointTs: 449348940019007493, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348940019007493, checkpointTs: 449348940019007493, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348940019007492, checkpointTs: 449348940019007492, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348940019007492,"min-table-barrier-ts":449348940019007493,"admin-job-type":0} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 + grep -q 'failed to get info:' {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0030d2990 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348940019007492 MinTableBarrierTs:449348940019007493 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348940019007493, checkpointTs: 449348940019007493, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348940019007493, checkpointTs: 449348940019007493, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348940019007492, checkpointTs: 449348940019007492, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348940019007492,"min-table-barrier-ts":449348940019007493,"admin-job-type":0} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 + grep -q 'etcd info' {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2024/04/26 19:16:13.506 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] [2024/04/26 19:16:13.684 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:16:13.696 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_claim_check/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:13 CST 2024] <<<<<< run test case canal_json_claim_check success! >>>>>> [2024/04/26 19:16:13.802 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] [2024/04/26 19:16:13.908 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.cli.2553.out cli tso query --pd=http://127.0.0.1:2379 [2024/04/26 19:16:13.976 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:16:13.980 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLse609701c_803c_4845_a7a5_6ac04607b63a"] [2024/04/26 19:16:13.984 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:16:14.089 +08:00] [INFO] [main.go:178] ["72 insert success: 1200"] [2024/04/26 19:16:14.172 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs2bd9fad1_f38c_4f93_b724_8c64ea501355"] check data successfully wait process cdc.test exit for 1-th time... check diff failed 15-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully [2024/04/26 19:16:14.370 +08:00] [INFO] [main.go:178] ["73 insert success: 1200"] [2024/04/26 19:16:14.387 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:16:14.682 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:14.700 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:16:14.701 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:14.704 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:16:14.709 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687afbe00018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:7368, start at 2024-04-26 19:16:13.726387143 +0800 CST m=+5.238914087 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:13.734 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:13.738 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:13.738 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... "is-owner": true, run task successfully [2024/04/26 19:16:14.775 +08:00] [INFO] [main.go:178] ["72 insert success: 1300"] [2024/04/26 19:16:14.793 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:16:14.805 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:16:14.902 +08:00] [INFO] [main.go:178] ["73 insert success: 1300"] [2024/04/26 19:16:14.978 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] wait process cdc.test exit for 1-th time... table ddl_manager.finish_mark not exists for 53-th check, retry later Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:15 CST 2024] <<<<<< run test case server_config_compatibility success! >>>>>> [2024/04/26 19:16:15.481 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] + set +x + tso='449348940312084481 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348940312084481 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:16:15 CST 2024] <<<<<< START cdc server in changefeed_fast_fail case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedFastFailError=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.25982600.out server --log-file /tmp/tidb_cdc_test/changefeed_fast_fail/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_fast_fail/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:16:15.575 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:15.580 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:16:15.585 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:16:15.671 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:16:15.771 +08:00] [INFO] [main.go:178] ["72 insert success: 1400"] [2024/04/26 19:16:15.792 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:16:15.796 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] wait process cdc.test exit for 3-th time... table consistent_replicate_gbk.GBKTABLE exists table consistent_replicate_gbk.check1 not exists for 1-th check, retry later [2024/04/26 19:16:15.911 +08:00] [INFO] [main.go:178] ["73 insert success: 1400"] check diff failed 16-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/open_protocol_claim_check/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:16 CST 2024] <<<<<< run test case open_protocol_claim_check success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:16.111 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:16.171 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:16.175 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:16.187 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:16.214 +08:00] [INFO] [main.go:178] ["72 insert success: 1500"] [2024/04/26 19:16:16.283 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:16.311 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] owner pid: 3301 owner id a5d76f32-b780-4b56-98d8-b96bcec2b84a [Fri Apr 26 19:16:16 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.34103412.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:16:16 CST 2024] <<<<<< run test case owner_resign success! >>>>>> [2024/04/26 19:16:15.933 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark1000] [2024/04/26 19:16:15.933 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark1000] [2024/04/26 19:16:16.484 +08:00] [INFO] [main.go:178] ["73 insert success: 1500"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687afbe00018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:7368, start at 2024-04-26 19:16:13.726387143 +0800 CST m=+5.238914087 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:13.734 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:13.738 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:13.738 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687afccc0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:7449, start at 2024-04-26 19:16:13.792477438 +0800 CST m=+5.240822317 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:13.799 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:13.796 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:13.796 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... [2024/04/26 19:16:16.681 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:16.772 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:16.774 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:16.777 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:16.789 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs8dab9a00_76d8_4c4b_8b22_780fa5b1267c"] [2024/04/26 19:16:16.796 +08:00] [INFO] [main.go:178] ["72 insert success: 1600"] TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_sequence/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/04/26 19:16:16.869 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:16.905 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] table ddl_manager.finish_mark not exists for 54-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 1-th time, retry later [2024/04/26 19:16:17.396 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:17.493 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:17.495 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:17.504 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:17.510 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:16:17.517 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:16:17.576 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:17.590 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:17.797 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] table consistent_replicate_gbk.check1 exists check diff successfully check diff failed 17-th time, retry later [2024/04/26 19:16:17.977 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:18.073 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:18.078 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:16:18.084 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] wait process cdc.test exit for 1-th time... [2024/04/26 19:16:17.933 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark1000] [2024/04/26 19:16:18.181 +08:00] [INFO] [main.go:835] ["running ddl test: testMultiDDLs75c93c48_ed5d_4ff9_aada_aa139ec23a6b"] [2024/04/26 19:16:18.193 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:18.195 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] [2024/04/26 19:16:18.214 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:18.404 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] wait process cdc.test exit for 2-th time... [2024/04/26 19:16:18.497 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:18.585 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:18.585 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:18.599 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:18.693 +08:00] [INFO] [main.go:178] ["72 insert success: 100"] [2024/04/26 19:16:18.704 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:18.704 +08:00] [INFO] [main.go:178] ["73 insert success: 100"] [2024/04/26 19:16:18.708 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] check diff failed 2-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:18 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c3bad39e-c09f-4aac-b0e1-f11762f667ce {"id":"c3bad39e-c09f-4aac-b0e1-f11762f667ce","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e883201 c3bad39e-c09f-4aac-b0e1-f11762f667ce /tidb/cdc/default/default/upstream/7362132981741967249 {"id":7362132981741967249,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c3bad39e-c09f-4aac-b0e1-f11762f667ce {"id":"c3bad39e-c09f-4aac-b0e1-f11762f667ce","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e883201 c3bad39e-c09f-4aac-b0e1-f11762f667ce /tidb/cdc/default/default/upstream/7362132981741967249 {"id":7362132981741967249,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c3bad39e-c09f-4aac-b0e1-f11762f667ce {"id":"c3bad39e-c09f-4aac-b0e1-f11762f667ce","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e883201 c3bad39e-c09f-4aac-b0e1-f11762f667ce /tidb/cdc/default/default/upstream/7362132981741967249 {"id":7362132981741967249,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_fast_fail.cli.2651.out cli changefeed create --start-ts=449348940312084481 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' -c changefeed-fast-fail + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8715.out cli tso query --pd=http://127.0.0.1:2379 [2024/04/26 19:16:18.932 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark2000] [2024/04/26 19:16:18.932 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark2000] [2024/04/26 19:16:18.779 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:18.908 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] [2024/04/26 19:16:18.992 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] Create changefeed successfully! ID: changefeed-fast-fail Info: {"upstream_id":7362132981741967249,"namespace":"default","id":"changefeed-fast-fail","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:16:18.960473571+08:00","start_ts":449348940312084481,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348940312084481,"checkpoint_ts":449348940312084481,"checkpoint_time":"2024-04-26 19:16:13.920"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2024/04/26 19:16:19.074 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:19.089 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:16:19.104 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:19.179 +08:00] [INFO] [main.go:178] ["72 insert success: 200"] [2024/04/26 19:16:19.192 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:19.274 +08:00] [INFO] [main.go:178] ["73 insert success: 200"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:19 CST 2024] <<<<<< START cdc server in consistent_replicate_gbk case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_gbk.56985700.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_storage_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:19 CST 2024] <<<<<< run test case canal_json_storage_basic success! >>>>>> [2024/04/26 19:16:19.282 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:19.298 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:19.498 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:19 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003859710 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348941329989641 MinTableBarrierTs:449348941592133641 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ed8eefcc-386c-4fef-b94d-f89dece6354c {"id":"ed8eefcc-386c-4fef-b94d-f89dece6354c","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a93 ed8eefcc-386c-4fef-b94d-f89dece6354c /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348941329989641,"min-table-barrier-ts":449348941592133641,"admin-job-type":0} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed8eefcc-386c-4fef-b94d-f89dece6354c/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003859710 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348941329989641 MinTableBarrierTs:449348941592133641 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ed8eefcc-386c-4fef-b94d-f89dece6354c {"id":"ed8eefcc-386c-4fef-b94d-f89dece6354c","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a93 ed8eefcc-386c-4fef-b94d-f89dece6354c /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348941329989641,"min-table-barrier-ts":449348941592133641,"admin-job-type":0} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed8eefcc-386c-4fef-b94d-f89dece6354c/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003859710 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348941329989641 MinTableBarrierTs:449348941592133641 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ed8eefcc-386c-4fef-b94d-f89dece6354c {"id":"ed8eefcc-386c-4fef-b94d-f89dece6354c","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a93 ed8eefcc-386c-4fef-b94d-f89dece6354c /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348941329989641,"min-table-barrier-ts":449348941592133641,"admin-job-type":0} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed8eefcc-386c-4fef-b94d-f89dece6354c/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8301' 2>&1 | grep -v "a5d76f32-b780-4b56-98d8-b96bcec2b84a" | grep id table ddl_manager.finish_mark not exists for 55-th check, retry later [2024/04/26 19:16:19.587 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:19.687 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:19.704 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:16:19.783 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:19.798 +08:00] [INFO] [main.go:178] ["72 insert success: 300"] [2024/04/26 19:16:19.880 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:19.904 +08:00] [INFO] [main.go:178] ["73 insert success: 300"] [2024/04/26 19:16:19.973 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:19.987 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:19.946 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark1000] [2024/04/26 19:16:19.964 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark1000] [ts=449348941831995398] [2024/04/26 19:16:19.964 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark1000] [2024/04/26 19:16:19.964 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348941831995398] [2024/04/26 19:16:19.970 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:20.018 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:20.024 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:20.030 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] + set +x check_changefeed_state http://127.0.0.1:2379 changefeed-fast-fail failed ErrStartTsBeforeGC + endpoints=http://127.0.0.1:2379 + changefeed_id=changefeed-fast-fail + expected_state=failed + error_msg=ErrStartTsBeforeGC + tls_dir=ErrStartTsBeforeGC + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c changefeed-fast-fail -s "cluster-id": "default" "id": "ed8eefcc-386c-4fef-b94d-f89dece6354c", "cluster-id": "default" run task successfully [2024/04/26 19:16:20.371 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] [2024/04/26 19:16:20.409 +08:00] [INFO] [main.go:178] ["72 insert success: 400"] [2024/04/26 19:16:20.482 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] check diff failed 18-th time, retry later + info='{ "upstream_id": 7362132981741967249, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:16:18.969539296+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' + echo '{ "upstream_id": 7362132981741967249, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:16:18.969539296+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' { "upstream_id": 7362132981741967249, "namespace": "default", "id": "changefeed-fast-fail", "state": "failed", "checkpoint_tso": 0, "checkpoint_time": "0001-01-01 00:00:00.000", "error": { "time": "2024-04-26T19:16:18.969539296+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7362132981741967249, '"namespace":' '"default",' '"id":' '"changefeed-fast-fail",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-04-26T19:16:18.969539296+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7362132981741967249, '"namespace":' '"default",' '"id":' '"changefeed-fast-fail",' '"state":' '"failed",' '"checkpoint_tso":' 0, '"checkpoint_time":' '"0001-01-01' '00:00:00.000",' '"error":' '{' '"time":' '"2024-04-26T19:16:18.969539296+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError' + [[ ! [CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError =~ ErrStartTsBeforeGC ]] run task successfully + set +x + tso='449348941679951876 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348941679951876 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:16:20 CST 2024] <<<<<< START cdc server in ddl_sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.87578759.out server --log-file /tmp/tidb_cdc_test/ddl_sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_sequence/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:16:20.682 +08:00] [INFO] [main.go:178] ["73 insert success: 400"] [2024/04/26 19:16:20.775 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:20.797 +08:00] [INFO] [main.go:178] ["73 insert success: 1000"] [2024/04/26 19:16:21.013 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:21.028 +08:00] [INFO] [main.go:178] ["72 insert success: 500"] [2024/04/26 19:16:21.085 +08:00] [INFO] [main.go:178] ["72 insert success: 1100"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:20.932 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark2000] check diff failed 3-th time, retry later [2024/04/26 19:16:21.125 +08:00] [INFO] [main.go:178] ["73 insert success: 500"] [2024/04/26 19:16:21.148 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:21.175 +08:00] [INFO] [main.go:178] ["73 insert success: 1100"] [2024/04/26 19:16:21.302 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:21.395 +08:00] [INFO] [main.go:178] ["72 insert success: 600"] table ddl_manager.finish_mark not exists for 56-th check, retry later [2024/04/26 19:16:21.478 +08:00] [INFO] [main.go:178] ["73 insert success: 600"] [2024/04/26 19:16:21.498 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] [2024/04/26 19:16:21.584 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] [2024/04/26 19:16:21.680 +08:00] [INFO] [main.go:178] ["72 insert success: 700"] [2024/04/26 19:16:21.783 +08:00] [INFO] [main.go:178] ["73 insert success: 700"] [2024/04/26 19:16:21.804 +08:00] [INFO] [main.go:178] ["73 insert success: 900"] [2024/04/26 19:16:21.869 +08:00] [INFO] [main.go:178] ["72 insert success: 1000"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/canal_json_storage_partition_table/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:21 CST 2024] <<<<<< run test case canal_json_storage_partition_table success! >>>>>> wait process cdc.test exit for 1-th time... [2024/04/26 19:16:21.966 +08:00] [INFO] [main.go:178] ["72 insert success: 800"] [2024/04/26 19:16:22.071 +08:00] [INFO] [main.go:178] ["73 insert success: 800"] wait process cdc.test exit for 2-th time... capture_id: ed8eefcc-386c-4fef-b94d-f89dece6354c [2024/04/26 19:16:22.252 +08:00] [INFO] [main.go:178] ["72 insert success: 900"] check diff failed 19-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:22 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ac815180-bb59-4493-9d6d-50681de8c5c0 {UpstreamID:7362132960996133459 Namespace:default ID:ac815180-bb59-4493-9d6d-50681de8c5c0 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:16:11.829709858 +0800 CST StartTs:449348939721736197 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031b3710 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348939761057794} {CheckpointTs:449348942238318598 MinTableBarrierTs:449348942238318598 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348942238318598, checkpointTs: 449348942238318598, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348942238318598, checkpointTs: 449348942238318598, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e0556a5-9cce-4ebd-9808-f0840c610be6 {"id":"8e0556a5-9cce-4ebd-9808-f0840c610be6","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e6ca473 8e0556a5-9cce-4ebd-9808-f0840c610be6 /tidb/cdc/default/default/changefeed/info/ac815180-bb59-4493-9d6d-50681de8c5c0 {"upstream-id":7362132960996133459,"namespace":"default","changefeed-id":"ac815180-bb59-4493-9d6d-50681de8c5c0","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:16:11.829709858+08:00","start-ts":449348939721736197,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348939761057794} /tidb/cdc/default/default/changefeed/status/ac815180-bb59-4493-9d6d-50681de8c5c0 {"checkpoint-ts":449348942238318598,"min-table-barrier-ts":449348942238318598,"admin-job-type":0} /tidb/cdc/default/default/task/position/8e0556a5-9cce-4ebd-9808-f0840c610be6/ac815180-bb59-4493-9d6d-50681de8c5c0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132960996133459 {"id":7362132960996133459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ac815180-bb59-4493-9d6d-50681de8c5c0 {UpstreamID:7362132960996133459 Namespace:default ID:ac815180-bb59-4493-9d6d-50681de8c5c0 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:16:11.829709858 +0800 CST StartTs:449348939721736197 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031b3710 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348939761057794} {CheckpointTs:449348942238318598 MinTableBarrierTs:449348942238318598 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348942238318598, checkpointTs: 449348942238318598, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348942238318598, checkpointTs: 449348942238318598, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e0556a5-9cce-4ebd-9808-f0840c610be6 {"id":"8e0556a5-9cce-4ebd-9808-f0840c610be6","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e6ca473 8e0556a5-9cce-4ebd-9808-f0840c610be6 /tidb/cdc/default/default/changefeed/info/ac815180-bb59-4493-9d6d-50681de8c5c0 {"upstream-id":7362132960996133459,"namespace":"default","changefeed-id":"ac815180-bb59-4493-9d6d-50681de8c5c0","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:16:11.829709858+08:00","start-ts":449348939721736197,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348939761057794} /tidb/cdc/default/default/changefeed/status/ac815180-bb59-4493-9d6d-50681de8c5c0 {"checkpoint-ts":449348942238318598,"min-table-barrier-ts":449348942238318598,"admin-job-type":0} /tidb/cdc/default/default/task/position/8e0556a5-9cce-4ebd-9808-f0840c610be6/ac815180-bb59-4493-9d6d-50681de8c5c0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132960996133459 {"id":7362132960996133459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ac815180-bb59-4493-9d6d-50681de8c5c0 {UpstreamID:7362132960996133459 Namespace:default ID:ac815180-bb59-4493-9d6d-50681de8c5c0 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:16:11.829709858 +0800 CST StartTs:449348939721736197 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031b3710 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348939761057794} {CheckpointTs:449348942238318598 MinTableBarrierTs:449348942238318598 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348942238318598, checkpointTs: 449348942238318598, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348942238318598, checkpointTs: 449348942238318598, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8e0556a5-9cce-4ebd-9808-f0840c610be6 {"id":"8e0556a5-9cce-4ebd-9808-f0840c610be6","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e6ca473 8e0556a5-9cce-4ebd-9808-f0840c610be6 /tidb/cdc/default/default/changefeed/info/ac815180-bb59-4493-9d6d-50681de8c5c0 {"upstream-id":7362132960996133459,"namespace":"default","changefeed-id":"ac815180-bb59-4493-9d6d-50681de8c5c0","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:16:11.829709858+08:00","start-ts":449348939721736197,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348939761057794} /tidb/cdc/default/default/changefeed/status/ac815180-bb59-4493-9d6d-50681de8c5c0 {"checkpoint-ts":449348942238318598,"min-table-barrier-ts":449348942238318598,"admin-job-type":0} /tidb/cdc/default/default/task/position/8e0556a5-9cce-4ebd-9808-f0840c610be6/ac815180-bb59-4493-9d6d-50681de8c5c0 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132960996133459 {"id":7362132960996133459,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b57b4001c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:3984, start at 2024-04-26 19:16:19.614840982 +0800 CST m=+5.506961009 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:19.622 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:19.615 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:19.615 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b57b4001c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:3984, start at 2024-04-26 19:16:19.614840982 +0800 CST m=+5.506961009 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:19.622 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:19.615 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:19.615 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b59400014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:4070, start at 2024-04-26 19:16:19.699468157 +0800 CST m=+5.542990855 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:19.707 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:19.664 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:19.664 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_add_table/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/batch_add_table/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:22 CST 2024] <<<<<< run test case changefeed_fast_fail success! >>>>>> [2024/04/26 19:16:22.882 +08:00] [INFO] [main.go:812] ["testMultiDDLs take %v1m21.223449246s"] [2024/04/26 19:16:22.930 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark3000] [2024/04/26 19:16:22.930 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark3000] [2024/04/26 19:16:23.022 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark2000] check diff failed 4-th time, retry later table ddl_manager.finish_mark not exists for 57-th check, retry later [2024/04/26 19:16:23.393 +08:00] [INFO] [main.go:74] ["DefaultValue integration tests take 1m21.73459832s"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_1 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:23 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b313eee-01de-4e0a-ae81-c26dfdf9d3aa {"id":"9b313eee-01de-4e0a-ae81-c26dfdf9d3aa","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e98a9f6 9b313eee-01de-4e0a-ae81-c26dfdf9d3aa /tidb/cdc/default/default/upstream/7362133002409550479 {"id":7362133002409550479,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b313eee-01de-4e0a-ae81-c26dfdf9d3aa {"id":"9b313eee-01de-4e0a-ae81-c26dfdf9d3aa","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e98a9f6 9b313eee-01de-4e0a-ae81-c26dfdf9d3aa /tidb/cdc/default/default/upstream/7362133002409550479 {"id":7362133002409550479,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9b313eee-01de-4e0a-ae81-c26dfdf9d3aa {"id":"9b313eee-01de-4e0a-ae81-c26dfdf9d3aa","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1e98a9f6 9b313eee-01de-4e0a-ae81-c26dfdf9d3aa /tidb/cdc/default/default/upstream/7362133002409550479 {"id":7362133002409550479,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_sequence.cli.8804.out cli changefeed create --start-ts=449348941679951876 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 5c771275-9a0b-444f-b9d3-17a4ded9635f Info: {"upstream_id":7362133002409550479,"namespace":"default","id":"5c771275-9a0b-444f-b9d3-17a4ded9635f","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:16:24.058408634+08:00","start_ts":449348941679951876,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348941679951876,"checkpoint_ts":449348941679951876,"checkpoint_time":"2024-04-26 19:16:19.138"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... ./t/alter_table.test: ok! 70 test cases passed, take time 20.972747425 s time="2024-04-26T19:16:23+08:00" level=info msg="run test [alter_table] ok" time="2024-04-26T19:16:23+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0000ca468 {{} {} 0} {0 0} [0xc000384000] map[] 0 1 0xc00003e960 false map[0xc000384000:map[0xc000384000:true]] map[] 0 0 0 0 0 0 0 0 0x4914e0}" check diff failed 20-th time, retry later [2024/04/26 19:16:24.930 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] check diff failed 5-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_tables_ddl/run.sh using Sink-Type: mysql... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 [2024/04/26 19:16:25.036 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark2000] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_big_messages/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:24 CST 2024] <<<<<< run test case kafka_big_messages success! >>>>>> table ddl_manager.finish_mark not exists for 58-th check, retry later You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend + set +x 0 [Fri Apr 26 19:16:24 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.35413543.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_cleanup_stale_tasks.server3.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_cleanup_stale_tasks.server3 --cluster-id default --addr 127.0.0.1:8302 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b987c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:3429, start at 2024-04-26 19:16:23.715340608 +0800 CST m=+8.712102312 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:23.734 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:23.711 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:23.711 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b987c0002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:3429, start at 2024-04-26 19:16:23.715340608 +0800 CST m=+8.712102312 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:23.734 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:23.711 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:23.711 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687b98140007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:3465, start at 2024-04-26 19:16:23.700805703 +0800 CST m=+8.535214905 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:23.725 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:23.735 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:23.735 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/region_merge/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/region_merge/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/region_merge/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/04/26 19:16:25.630 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark4000] [2024/04/26 19:16:25.630 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark4000] Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide table mark.finish_mark_1 not exists for 2-th check, retry later check diff failed 21-th time, retry later * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17C9D052B78D1608 < X-Xss-Protection: 1; mode=block < Date: Fri, 26 Apr 2024 11:16:26 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... [2024/04/26 19:16:27.018 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] [2024/04/26 19:16:27.052 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark2000] [2024/04/26 19:16:27.227 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark2000] [ts=449348943614574600] [2024/04/26 19:16:27.227 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark2000] [2024/04/26 19:16:27.227 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348943614574600] [2024/04/26 19:16:27.318 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:27.324 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:27.331 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:27.422 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] table ddl_sequence.finish_mark not exists for 1-th check, retry later check diff failed 6-th time, retry later [2024/04/26 19:16:27.631 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] table ddl_manager.finish_mark not exists for 59-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:27 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002bfe630 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348942116421640 MinTableBarrierTs:449348942116421642 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/234b32df-214b-4cb3-983f-7e57c51ff891 {"id":"234b32df-214b-4cb3-983f-7e57c51ff891","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ed8eefcc-386c-4fef-b94d-f89dece6354c {"id":"ed8eefcc-386c-4fef-b94d-f89dece6354c","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a93 ed8eefcc-386c-4fef-b94d-f89dece6354c /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2acf 234b32df-214b-4cb3-983f-7e57c51ff891 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348942116421640,"min-table-barrier-ts":449348942116421642,"admin-job-type":0} /tidb/cdc/default/default/task/position/234b32df-214b-4cb3-983f-7e57c51ff891/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed8eefcc-386c-4fef-b94d-f89dece6354c/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002bfe630 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348942116421640 MinTableBarrierTs:449348942116421642 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/234b32df-214b-4cb3-983f-7e57c51ff891 {"id":"234b32df-214b-4cb3-983f-7e57c51ff891","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ed8eefcc-386c-4fef-b94d-f89dece6354c {"id":"ed8eefcc-386c-4fef-b94d-f89dece6354c","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a93 ed8eefcc-386c-4fef-b94d-f89dece6354c /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2acf 234b32df-214b-4cb3-983f-7e57c51ff891 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348942116421640,"min-table-barrier-ts":449348942116421642,"admin-job-type":0} /tidb/cdc/default/default/task/position/234b32df-214b-4cb3-983f-7e57c51ff891/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed8eefcc-386c-4fef-b94d-f89dece6354c/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002bfe630 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348942116421640 MinTableBarrierTs:449348942116421642 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/234b32df-214b-4cb3-983f-7e57c51ff891 {"id":"234b32df-214b-4cb3-983f-7e57c51ff891","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/a5d76f32-b780-4b56-98d8-b96bcec2b84a {"id":"a5d76f32-b780-4b56-98d8-b96bcec2b84a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/ed8eefcc-386c-4fef-b94d-f89dece6354c {"id":"ed8eefcc-386c-4fef-b94d-f89dece6354c","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a58 a5d76f32-b780-4b56-98d8-b96bcec2b84a /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2a93 ed8eefcc-386c-4fef-b94d-f89dece6354c /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2acf 234b32df-214b-4cb3-983f-7e57c51ff891 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348942116421640,"min-table-barrier-ts":449348942116421642,"admin-job-type":0} /tidb/cdc/default/default/task/position/234b32df-214b-4cb3-983f-7e57c51ff891/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/a5d76f32-b780-4b56-98d8-b96bcec2b84a/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/ed8eefcc-386c-4fef-b94d-f89dece6354c/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' table mark.finish_mark_1 not exists for 3-th check, retry later run task failed 1-th time, retry later check diff failed 22-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_etcd_worker_delay/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/04/26 19:16:29.033 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] check_redo_resolved_ts ac815180-bb59-4493-9d6d-50681de8c5c0 449348943955361798 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/ac815180-bb59-4493-9d6d-50681de8c5c0/meta global resolved ts 449348943824289793 not forward to 449348943955361798 run task failed 1-th time, retry later [Fri Apr 26 19:16:28 CST 2024] <<<<<< START cdc server in batch_add_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.53355337.out server --log-file /tmp/tidb_cdc_test/batch_add_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_add_table/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully [2024/04/26 19:16:29.431 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark5000] [2024/04/26 19:16:29.431 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark5000] table ddl_manager.finish_mark not exists for 60-th check, retry later [2024/04/26 19:16:29.736 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] [Fri Apr 26 19:16:29 CST 2024] <<<<<< START cdc server in region_merge case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.46944696.out server --log-file /tmp/tidb_cdc_test/region_merge/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/region_merge/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... table ddl_sequence.finish_mark not exists for 2-th check, retry later check_redo_resolved_ts ac815180-bb59-4493-9d6d-50681de8c5c0 449348943955361798 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_gbk/cdc_data/redo/ac815180-bb59-4493-9d6d-50681de8c5c0/meta table mark.finish_mark_1 exists table mark.finish_mark_2 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/multi_tables_ddl Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... run task successfully cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' run task failed 2-th time, retry later check diff failed 23-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:30 CST 2024] <<<<<< run test case changefeed_pause_resume success! >>>>>> wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > [2024/04/26 19:16:31.046 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:31 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d81cd08a-15f0-480f-b1b6-5001aaf01078 {"id":"d81cd08a-15f0-480f-b1b6-5001aaf01078","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb09834 d81cd08a-15f0-480f-b1b6-5001aaf01078 /tidb/cdc/default/default/upstream/7362133025862491743 {"id":7362133025862491743,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d81cd08a-15f0-480f-b1b6-5001aaf01078 {"id":"d81cd08a-15f0-480f-b1b6-5001aaf01078","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb09834 d81cd08a-15f0-480f-b1b6-5001aaf01078 /tidb/cdc/default/default/upstream/7362133025862491743 {"id":7362133025862491743,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d81cd08a-15f0-480f-b1b6-5001aaf01078 {"id":"d81cd08a-15f0-480f-b1b6-5001aaf01078","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eb09834 d81cd08a-15f0-480f-b1b6-5001aaf01078 /tidb/cdc/default/default/upstream/7362133025862491743 {"id":7362133025862491743,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_add_table.cli.5386.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ [2024/04/26 19:16:31.431 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] table ddl_manager.finish_mark not exists for 61-th check, retry later Create changefeed successfully! ID: 5dd3b67a-3207-4e13-b2c1-ca23603f58cf Info: {"upstream_id":7362133025862491743,"namespace":"default","id":"5dd3b67a-3207-4e13-b2c1-ca23603f58cf","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:16:31.589089748+08:00","start_ts":449348944911663106,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348944911663106,"checkpoint_ts":449348944911663106,"checkpoint_time":"2024-04-26 19:16:31.466"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [2024/04/26 19:16:31.628 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/04/26 19:16:31.629 +08:00] [INFO] [s3.go:407] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/04/26 19:16:31.633 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/04/26 19:16:31.633 +08:00] [INFO] [redo.go:163] ["apply redo log starts"] [checkpointTs=449348942487355421] [resolvedTs=449348944348839937] [2024/04/26 19:16:31.634 +08:00] [INFO] [s3.go:407] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/04/26 19:16:31.634 +08:00] [INFO] [s3.go:407] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/04/26 19:16:31.634 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:16:31.634 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:16:31.637 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=3.765292ms] [2024/04/26 19:16:31.639 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=5.592459ms] [2024/04/26 19:16:31.639 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:16:31.642 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/04/26 19:16:31.642 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/04/26 19:16:31.643 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:16:31.643 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:16:31.647 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CONLY_FULL_GROUP_BY%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:16:31.649 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/04/26 19:16:31.649 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449348942487355417,\"CommitTs\":449348942487355421,\"Query\":\"create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_gbk\",\"tbl-name\":\"GBKTABLE2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:16:31.650 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449348942487355417,\"CommitTs\":449348942487355421,\"Query\":\"create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_gbk\",\"tbl-name\":\"GBKTABLE2\",\"tbl-id\":108,\"is-partition\":false},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":3,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:16:31.651 +08:00] [INFO] [mysql_ddl_sink.go:119] ["Execute DDL failed, but error can be ignored"] [startTs=449348942487355417] [ddl="create table consistent_replicate_gbk.GBKTABLE2 like consistent_replicate_gbk.GBKTABLE"] [namespace=] [changefeed=] [error="Error 1050 (42S01): Table 'consistent_replicate_gbk.GBKTABLE2' already exists"] [2024/04/26 19:16:31.750 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] table ddl_sequence.finish_mark not exists for 3-th check, retry later [2024/04/26 19:16:31.852 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=449348944348839937] [2024/04/26 19:16:31.852 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=449348944348839937] table mark.finish_mark_2 not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/processor_etcd_worker_delay Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:16:31.953 +08:00] [INFO] [redo.go:219] ["apply redo log finishes"] [appliedLogCount=1] [appliedDDLCount=1] [currentCheckpoint=449348944348839937] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/04/26 19:16:31.953 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] Apply redo log successfully check diff successfully [Fri Apr 26 19:16:32 CST 2024] <<<<<< run test case consistent_replicate_gbk success! >>>>>> Exiting on signal: INTERRUPT [2024/04/26 19:16:32.328 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark6000] [2024/04/26 19:16:32.329 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark6000] + set +x check diff failed 24-th time, retry later [2024/04/26 19:16:33.060 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:32 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/725a348a-84b4-494f-9014-f60a3ea0c496 {"id":"725a348a-84b4-494f-9014-f60a3ea0c496","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ead6b11 725a348a-84b4-494f-9014-f60a3ea0c496 /tidb/cdc/default/default/upstream/7362133024542751020 {"id":7362133024542751020,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/725a348a-84b4-494f-9014-f60a3ea0c496 {"id":"725a348a-84b4-494f-9014-f60a3ea0c496","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ead6b11 725a348a-84b4-494f-9014-f60a3ea0c496 /tidb/cdc/default/default/upstream/7362133024542751020 {"id":7362133024542751020,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/725a348a-84b4-494f-9014-f60a3ea0c496 {"id":"725a348a-84b4-494f-9014-f60a3ea0c496","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1ead6b11 725a348a-84b4-494f-9014-f60a3ea0c496 /tidb/cdc/default/default/upstream/7362133024542751020 {"id":7362133024542751020,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.region_merge.cli.4731.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_compression/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:32 CST 2024] <<<<<< run test case kafka_compression success! >>>>>> [2024/04/26 19:16:33.444 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] table ddl_manager.finish_mark not exists for 62-th check, retry later table batch_add_table.finish_mark not exists for 1-th check, retry later table ddl_sequence.finish_mark not exists for 4-th check, retry later Create changefeed successfully! ID: cd55d0f6-543d-45d1-9624-b61c2b013fb0 Info: {"upstream_id":7362133024542751020,"namespace":"default","id":"cd55d0f6-543d-45d1-9624-b61c2b013fb0","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:16:33.605685147+08:00","start_ts":449348945421795331,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348945421795331,"checkpoint_ts":449348945421795331,"checkpoint_time":"2024-04-26 19:16:33.412"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table mark.finish_mark_2 not exists for 3-th check, retry later [2024/04/26 19:16:33.764 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:16:34.329 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release cdc.test cli capture list --server 'http://127.0.0.1:8302' 2>&1 | grep '"is-owner": true' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/batch_update_to_no_batch/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff successfully [2024/04/26 19:16:35.118 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark3000] [2024/04/26 19:16:35.135 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark3000] [ts=449348945370939397] [2024/04/26 19:16:35.135 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark3000] [2024/04/26 19:16:35.135 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348945370939397] [2024/04/26 19:16:35.138 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:35.221 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:35.226 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:35.230 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] wait process cdc.test exit for 1-th time... [2024/04/26 19:16:35.518 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] table ddl_sequence.finish_mark not exists for 5-th check, retry later + set +x table mark.finish_mark_2 not exists for 4-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_messages/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:35 CST 2024] <<<<<< run test case kafka_messages success! >>>>>> [2024/04/26 19:16:35.832 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] split_and_random_merge scale: 20 "is-owner": true, run task successfully nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table ddl_manager.finish_mark not exists for 63-th check, retry later table batch_add_table.finish_mark exists check diff successfully Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... [2024/04/26 19:16:36.030 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark7000] [2024/04/26 19:16:36.030 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark7000] wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:36.430 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] wait process cdc.test exit for 3-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:16:37 CST 2024] <<<<<< run test case clustered_index success! >>>>>> cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:37 CST 2024] <<<<<< run test case batch_add_table success! >>>>>> table ddl_sequence.finish_mark not exists for 6-th check, retry later [2024/04/26 19:16:37.531 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [2024/04/26 19:16:37.845 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] table ddl_manager.finish_mark not exists for 64-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:38.030 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] table mark.finish_mark_2 not exists for 5-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [2024/04/26 19:16:38.446 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] [2024/04/26 19:16:38.733 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark8000] [2024/04/26 19:16:38.733 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark8000] ./t/bigint.test: ok! 138 test cases passed, take time 12.423566512 s time="2024-04-26T19:16:36+08:00" level=info msg="run test [bigint] ok" time="2024-04-26T19:16:36+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc00030e188 {{} {} 0} {0 0} [0xc000384b40] map[] 0 1 0xc0003005a0 false map[0xc000384b40:map[0xc000384b40:true]] map[] 0 0 0 0 0 0 0 0 0x4914e0}" ./t/composite_index.test: ok! 12 test cases passed, take time 1.556063105 s time="2024-04-26T19:16:38+08:00" level=info msg="run test [composite_index] ok" time="2024-04-26T19:16:38+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000126188 {{} {} 0} {0 0} [0xc0000fe240] map[] 0 1 0xc00003eba0 false map[0xc0000fe240:map[0xc0000fe240:true]] map[] 0 0 0 0 0 0 0 0 0x4914e0}" ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 65-th check, retry later [2024/04/26 19:16:39.545 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [2024/04/26 19:16:39.858 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark4000] table ddl_sequence.finish_mark not exists for 7-th check, retry later [2024/04/26 19:16:39.919 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark4000] [ts=449348947101089797] [2024/04/26 19:16:39.919 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark4000] [2024/04/26 19:16:39.919 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348947101089797] [2024/04/26 19:16:39.921 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:39.925 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:39.930 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:39.935 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:40.043 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] table mark.finish_mark_2 not exists for 6-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ./t/concurrent_ddl.test: ok! 8 test cases passed, take time 1.027245523 s time="2024-04-26T19:16:40+08:00" level=info msg="run test [concurrent_ddl] ok" time="2024-04-26T19:16:40+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000498058 {{} {} 0} {0 0} [0xc0003c2090] map[] 0 1 0xc0004801e0 false map[0xc0003c2090:map[0xc0003c2090:true]] map[] 0 0 0 0 0 0 0 0 0x4914e0}" ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:40.525 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/cli/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/04/26 19:16:40.733 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_sink_error_resume/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:40 CST 2024] <<<<<< run test case kafka_sink_error_resume success! >>>>>> table ddl_manager.finish_mark not exists for 66-th check, retry later [2024/04/26 19:16:41.618 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] table ddl_sequence.finish_mark not exists for 8-th check, retry later table mark.finish_mark_2 not exists for 7-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_nfs/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [2024/04/26 19:16:42.118 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] empty select id, val from test.availability1 where id=1 run task successfully test_owner_cleanup_stale_tasks pass VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ca6780014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:32562, start at 2024-04-26 19:16:41.011179218 +0800 CST m=+5.226504149 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:41.020 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:40.990 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:40.990 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ca6780014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:32562, start at 2024-04-26 19:16:41.011179218 +0800 CST m=+5.226504149 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:41.020 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:40.990 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:40.990 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ca7300015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-9rmxk, pid:32626, start at 2024-04-26 19:16:41.078959166 +0800 CST m=+5.230074994 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:41.087 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:41.086 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:41.086 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_tables_ddl/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } *************************** 1. row *************************** count(distinct region_id): 1 [2024/04/26 19:16:42.530 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark9000] [2024/04/26 19:16:42.530 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark9000] [2024/04/26 19:16:42.539 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] wait process cdc.test exit for 1-th time... [2024/04/26 19:16:42.929 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] start tidb cluster in /tmp/tidb_cdc_test/batch_update_to_no_batch Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/mq_sink_lost_callback/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:43 CST 2024] <<<<<< run test case mq_sink_lost_callback success! >>>>>> table ddl_manager.finish_mark not exists for 67-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_owner_retryable_error [Fri Apr 26 19:16:43 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/capture/capture-campaign-compacted-error=1*return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.37843786.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_retryable_error.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_retryable_error.server1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_sequence.finish_mark exists [2024/04/26 19:16:43.637 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] table mark.finish_mark_2 not exists for 8-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/cli Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [Fri Apr 26 19:16:44 CST 2024] <<<<<< START cdc server in multi_tables_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_tables_ddl.3394133943.out server --log-file /tmp/tidb_cdc_test/multi_tables_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_tables_ddl/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:16:44.138 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_nfs Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:44.530 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:16:44.552 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] check diff successfully Verifying downstream PD is started... [2024/04/26 19:16:44.944 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] wait process cdc.test exit for 1-th time... [2024/04/26 19:16:45.330 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark10000] [2024/04/26 19:16:45.650 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] wait process cdc.test exit for 2-th time... table mark.finish_mark_2 not exists for 9-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/mq_sink_dispatcher/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:45 CST 2024] <<<<<< run test case mq_sink_dispatcher success! >>>>>> table ddl_manager.finish_mark not exists for 68-th check, retry later [2024/04/26 19:16:46.151 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:16:46 CST 2024] <<<<<< run test case ddl_sequence success! >>>>>> [2024/04/26 19:16:46.544 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) split_and_random_merge scale: 40 [2024/04/26 19:16:46.564 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:46 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037cfa70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348947634290693 MinTableBarrierTs:449348948682866691 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c915f88-1622-4c49-9a33-f0d9519a9df3 {"id":"8c915f88-1622-4c49-9a33-f0d9519a9df3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b36 8c915f88-1622-4c49-9a33-f0d9519a9df3 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348947634290693,"min-table-barrier-ts":449348948682866691,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c915f88-1622-4c49-9a33-f0d9519a9df3/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037cfa70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348947634290693 MinTableBarrierTs:449348948682866691 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c915f88-1622-4c49-9a33-f0d9519a9df3 {"id":"8c915f88-1622-4c49-9a33-f0d9519a9df3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b36 8c915f88-1622-4c49-9a33-f0d9519a9df3 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348947634290693,"min-table-barrier-ts":449348948682866691,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c915f88-1622-4c49-9a33-f0d9519a9df3/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037cfa70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348947634290693 MinTableBarrierTs:449348948682866691 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348947634290693, checkpointTs: 449348947634290693, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c915f88-1622-4c49-9a33-f0d9519a9df3 {"id":"8c915f88-1622-4c49-9a33-f0d9519a9df3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b36 8c915f88-1622-4c49-9a33-f0d9519a9df3 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348947634290693,"min-table-barrier-ts":449348948682866691,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c915f88-1622-4c49-9a33-f0d9519a9df3/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' ./t/date_formats.test: ok! 99 test cases passed, take time 5.314860475 s time="2024-04-26T19:16:46+08:00" level=info msg="run test [date_formats] ok" time="2024-04-26T19:16:46+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc000498278 {{} {} 0} {0 0} [0xc0003c2b40] map[] 0 1 0xc0003b6ba0 false map[0xc0003c2b40:map[0xc0003c2b40:true]] map[] 0 0 0 0 0 0 0 0 0x4914e0}" [2024/04/26 19:16:46.956 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] Verifying downstream PD is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/processor_resolved_ts_fallback/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_rocks/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:47 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ca705dad-4ab0-4100-a264-54c5029aef91 {"id":"ca705dad-4ab0-4100-a264-54c5029aef91","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eff69f1 ca705dad-4ab0-4100-a264-54c5029aef91 /tidb/cdc/default/default/upstream/7362133119017085058 {"id":7362133119017085058,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ca705dad-4ab0-4100-a264-54c5029aef91 {"id":"ca705dad-4ab0-4100-a264-54c5029aef91","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eff69f1 ca705dad-4ab0-4100-a264-54c5029aef91 /tidb/cdc/default/default/upstream/7362133119017085058 {"id":7362133119017085058,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ca705dad-4ab0-4100-a264-54c5029aef91 {"id":"ca705dad-4ab0-4100-a264-54c5029aef91","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1eff69f1 ca705dad-4ab0-4100-a264-54c5029aef91 /tidb/cdc/default/default/upstream/7362133119017085058 {"id":7362133119017085058,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: test-normal Info: {"upstream_id":7362133119017085058,"namespace":"default","id":"test-normal","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:16:47.468545966+08:00","start_ts":449348948247183361,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t1","multi_tables_ddl_test.t2","multi_tables_ddl_test.t3","multi_tables_ddl_test.t4","multi_tables_ddl_test.t1_7","multi_tables_ddl_test.t2_7","multi_tables_ddl_test.finish_mark"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348948247183361,"checkpoint_ts":449348948247183361,"checkpoint_time":"2024-04-26 19:16:44.190"} Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Create changefeed successfully! ID: test-error-1 Info: {"upstream_id":7362133119017085058,"namespace":"default","id":"test-error-1","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:16:47.618516929+08:00","start_ts":449348948247183361,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t5","multi_tables_ddl_test.t6","multi_tables_ddl_test.t7","multi_tables_ddl_test.t8"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348948247183361,"checkpoint_ts":449348948247183361,"checkpoint_time":"2024-04-26 19:16:44.190"} [2024/04/26 19:16:47.720 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark5000] [2024/04/26 19:16:47.827 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark5000] [ts=449348948962312197] [2024/04/26 19:16:47.827 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark5000] [2024/04/26 19:16:47.827 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348948962312197] [2024/04/26 19:16:47.830 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:47.922 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:47.928 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] Create changefeed successfully! ID: test-error-2 Info: {"upstream_id":7362133119017085058,"namespace":"default","id":"test-error-2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:16:47.768566518+08:00","start_ts":449348948247183361,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["multi_tables_ddl_test.t9","multi_tables_ddl_test.t10"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":true,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348948247183361,"checkpoint_ts":449348948247183361,"checkpoint_time":"2024-04-26 19:16:44.190"} table ddl_manager.finish_mark not exists for 69-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:48.019 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:48.019 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark10000] [2024/04/26 19:16:48.166 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] "is-owner": true, run task successfully table mark.finish_mark_2 not exists for 10-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore [2024/04/26 19:16:48.618 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:16:48.628 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:48.830 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark11000] [2024/04/26 19:16:49.018 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d12c80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:3545, start at 2024-04-26 19:16:47.934759047 +0800 CST m=+11.504707330 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:47.985 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:47.970 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:47.970 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d12c80008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:3545, start at 2024-04-26 19:16:47.934759047 +0800 CST m=+11.504707330 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:47.985 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:47.970 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:47.970 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d10e4000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:3578, start at 2024-04-26 19:16:47.838423786 +0800 CST m=+11.269874384 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:47.893 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:47.901 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:47.901 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_column_selector/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:49 CST 2024] <<<<<< run test case kafka_column_selector success! >>>>>> Logging trace to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_etcd_worker_delay/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } owner pid: 3789 owner id 8c915f88-1622-4c49-9a33-f0d9519a9df3 [Fri Apr 26 19:16:49 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/owner-run-with-error=1*return(true);github.com/pingcap/tiflow/cdc/capture/capture-resign-failed=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.38943896.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_owner_retryable_error.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_owner_retryable_error.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/processor_resolved_ts_fallback Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... start tidb cluster in /tmp/tidb_cdc_test/multi_rocks Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table ddl_manager.finish_mark not exists for 70-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_2 exists table mark.finish_mark_3 not exists for 1-th check, retry later [2024/04/26 19:16:50.020 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] [2024/04/26 19:16:50.318 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] [2024/04/26 19:16:50.638 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:50.720 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:16:51.033 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] [Fri Apr 26 19:16:51 CST 2024] <<<<<< START cdc server in processor_etcd_worker_delay case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/pkg/orchestrator/ProcessorEtcdDelay=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_etcd_worker_delay.48314833.out server --log-file /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_etcd_worker_delay/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table ddl_manager.finish_mark not exists for 71-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:51.732 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark12000] [2024/04/26 19:16:52.032 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] table mark.finish_mark_3 not exists for 2-th check, retry later [2024/04/26 19:16:52.330 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:52 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002db1680 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348950255992837 MinTableBarrierTs:449348950255992838 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/75af8e54-23fa-432d-aa0e-6a7f816c3d6b {"id":"75af8e54-23fa-432d-aa0e-6a7f816c3d6b","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8c915f88-1622-4c49-9a33-f0d9519a9df3 {"id":"8c915f88-1622-4c49-9a33-f0d9519a9df3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b36 8c915f88-1622-4c49-9a33-f0d9519a9df3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b6b 75af8e54-23fa-432d-aa0e-6a7f816c3d6b /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348950255992837,"min-table-barrier-ts":449348950255992838,"admin-job-type":0} /tidb/cdc/default/default/task/position/75af8e54-23fa-432d-aa0e-6a7f816c3d6b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8c915f88-1622-4c49-9a33-f0d9519a9df3/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002db1680 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348950255992837 MinTableBarrierTs:449348950255992838 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/75af8e54-23fa-432d-aa0e-6a7f816c3d6b {"id":"75af8e54-23fa-432d-aa0e-6a7f816c3d6b","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8c915f88-1622-4c49-9a33-f0d9519a9df3 {"id":"8c915f88-1622-4c49-9a33-f0d9519a9df3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b36 8c915f88-1622-4c49-9a33-f0d9519a9df3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b6b 75af8e54-23fa-432d-aa0e-6a7f816c3d6b /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348950255992837,"min-table-barrier-ts":449348950255992838,"admin-job-type":0} /tidb/cdc/default/default/task/position/75af8e54-23fa-432d-aa0e-6a7f816c3d6b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8c915f88-1622-4c49-9a33-f0d9519a9df3/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002db1680 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348950255992837 MinTableBarrierTs:449348950255992838 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/75af8e54-23fa-432d-aa0e-6a7f816c3d6b {"id":"75af8e54-23fa-432d-aa0e-6a7f816c3d6b","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/8c915f88-1622-4c49-9a33-f0d9519a9df3 {"id":"8c915f88-1622-4c49-9a33-f0d9519a9df3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b36 8c915f88-1622-4c49-9a33-f0d9519a9df3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2b6b 75af8e54-23fa-432d-aa0e-6a7f816c3d6b /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348950255992837,"min-table-barrier-ts":449348950255992838,"admin-job-type":0} /tidb/cdc/default/default/task/position/75af8e54-23fa-432d-aa0e-6a7f816c3d6b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/8c915f88-1622-4c49-9a33-f0d9519a9df3/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "8c915f88-1622-4c49-9a33-f0d9519a9df3" | grep id [2024/04/26 19:16:52.651 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:16:52.733 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore [2024/04/26 19:16:53.052 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) *************************** 1. row *************************** count(distinct region_id): 1 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_manager.finish_mark not exists for 72-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used Debugger for raftstore-v2 is used [2024/04/26 19:16:54.045 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] table big_txn.usertable1 exists table mark.finish_mark_3 not exists for 3-th check, retry later "id": "75af8e54-23fa-432d-aa0e-6a7f816c3d6b", "cluster-id": "default" "cluster-id": "default" run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_column_selector_avro/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:16:53 CST 2024] <<<<<< run test case kafka_column_selector_avro success! >>>>>> [2024/04/26 19:16:54.347 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] [2024/04/26 19:16:54.351 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark13000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:16:54 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec {"id":"b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f058dfa b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec /tidb/cdc/default/default/upstream/7362133121821191149 {"id":7362133121821191149,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec {"id":"b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f058dfa b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec /tidb/cdc/default/default/upstream/7362133121821191149 {"id":7362133121821191149,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec {"id":"b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f058dfa b7a8f21f-57cd-4a8e-b9d5-490d5972f7ec /tidb/cdc/default/default/upstream/7362133121821191149 {"id":7362133121821191149,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [2024/04/26 19:16:54.720 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:16:54.747 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark6000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:16:54.929 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark6000] [ts=449348950823534597] [2024/04/26 19:16:54.929 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark6000] [2024/04/26 19:16:54.929 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348950823534597] [2024/04/26 19:16:55.019 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:55.024 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:55.036 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:16:55.118 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] [2024/04/26 19:16:55.125 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:16:55.125 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark11000] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d73040013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:10123, start at 2024-04-26 19:16:54.129933304 +0800 CST m=+5.461975494 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:54.138 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:54.131 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:54.131 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) capture_id: 75af8e54-23fa-432d-aa0e-6a7f816c3d6b % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 35 100 35 0 0 26861 0 --:--:-- --:--:-- --:--:-- 35000 { "status": true, "message": "" }ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep '8c915f88-1622-4c49-9a33-f0d9519a9df3' 8c915f88-1622-4c49-9a33-f0d9519a9df3 run task successfully ps -C cdc.test -o pid= | awk '{print $1}' | wc -l | grep 2 2 run task successfully test_owner_retryable_error pass table ddl_manager.finish_mark not exists for 73-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d7e200002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:6799, start at 2024-04-26 19:16:54.794639223 +0800 CST m=+5.137776461 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:54.800 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:54.792 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:54.792 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d7e200002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:6799, start at 2024-04-26 19:16:54.794639223 +0800 CST m=+5.137776461 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:54.800 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:54.792 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:54.792 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d81140013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:6880, start at 2024-04-26 19:16:55.0120188 +0800 CST m=+5.308004134 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:55.020 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:54.981 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:54.981 +0800 All versions after safe point can be accessed. (DO NOT EDIT) wait process cdc.test exit for 1-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 4-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_nfs/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d690c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:3744, start at 2024-04-26 19:16:53.467181959 +0800 CST m=+5.441536458 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:53.480 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:53.443 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:53.443 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d690c0016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:3744, start at 2024-04-26 19:16:53.467181959 +0800 CST m=+5.441536458 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:53.480 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:53.443 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:53.443 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d6a900013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:3817, start at 2024-04-26 19:16:53.570653382 +0800 CST m=+5.445777025 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:53.578 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:53.540 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:53.540 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/batch_update_to_no_batch/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 2-th time... [2024/04/26 19:16:56.118 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] [2024/04/26 19:16:56.362 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] ./t/drop.test: ok! 48 test cases passed, take time 8.719864293 s time="2024-04-26T19:16:55+08:00" level=info msg="run test [drop] ok" time="2024-04-26T19:16:55+08:00" level=warning msg="Create new db&{{{} {} 0} 0xc0000ca408 {{} {} 0} {0 0} [0xc0003c2d80] map[] 0 1 0xc000503320 false map[0xc0003c2d80:map[0xc0003c2d80:true]] map[] 0 0 0 0 0 0 0 0 0x4914e0}" ./t/mysql_replace.test: ok! 16 test cases passed, take time 1.055809876 s time="2024-04-26T19:16:56+08:00" level=info msg="run test [mysql_replace] ok" Great, All tests passed mysqltest end cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_gap_between_watch_capture [Fri Apr 26 19:16:56 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/sleep-in-owner-tick=1*sleep(6000)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.40454047.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d73040013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:10123, start at 2024-04-26 19:16:54.129933304 +0800 CST m=+5.461975494 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:54.138 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:54.131 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:54.131 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687d73ac0007 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:10182, start at 2024-04-26 19:16:54.130905735 +0800 CST m=+5.406483639 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:54.139 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:54.123 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:54.123 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cli/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cli/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cli/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cli/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cli/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/04/26 19:16:56.835 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:16:57.125 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:16:57.218 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] table test.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 5-th check, retry later table ddl_manager.finish_mark not exists for 74-th check, retry later [2024/04/26 19:16:58.134 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] [2024/04/26 19:16:58.430 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] [2024/04/26 19:16:58.431 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark14000] split_and_random_merge scale: 80 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/resourcecontrol/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Fri Apr 26 19:16:57 CST 2024] <<<<<< START cdc server in consistent_replicate_nfs case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_nfs.81788180.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_nfs/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_nfs/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:16:58.848 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:16:59.138 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:16:59.234 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11446.out cli tso query --pd=http://127.0.0.1:2379 table test.finish_mark not exists for 2-th check, retry later table ddl_manager.finish_mark not exists for 75-th check, retry later [2024/04/26 19:17:00.147 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 6-th check, retry later [2024/04/26 19:17:00.443 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] [Fri Apr 26 19:17:00 CST 2024] <<<<<< START cdc server in batch_update_to_no_batch case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.50845086.out server --log-file /tmp/tidb_cdc_test/batch_update_to_no_batch/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/batch_update_to_no_batch/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='449348952216305665 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348952216305665 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [2024/04/26 19:17:00.862 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:01 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/360effe0-b0b2-4d7e-b570-d91a1003c701 {"id":"360effe0-b0b2-4d7e-b570-d91a1003c701","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f3777f3 360effe0-b0b2-4d7e-b570-d91a1003c701 /tidb/cdc/default/default/upstream/7362133172026260611 {"id":7362133172026260611,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/360effe0-b0b2-4d7e-b570-d91a1003c701 {"id":"360effe0-b0b2-4d7e-b570-d91a1003c701","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f3777f3 360effe0-b0b2-4d7e-b570-d91a1003c701 /tidb/cdc/default/default/upstream/7362133172026260611 {"id":7362133172026260611,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/360effe0-b0b2-4d7e-b570-d91a1003c701 {"id":"360effe0-b0b2-4d7e-b570-d91a1003c701","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f3777f3 360effe0-b0b2-4d7e-b570-d91a1003c701 /tidb/cdc/default/default/upstream/7362133172026260611 {"id":7362133172026260611,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:17:00 CST 2024] <<<<<< START cdc server in cli case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.1149611498.out server --log-file /tmp/tidb_cdc_test/cli/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cli/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:17:01.131 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark15000] [2024/04/26 19:17:01.151 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:01.247 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687dbab00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4, pid:4559, start at 2024-04-26 19:16:58.67687893 +0800 CST m=+5.657334253 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:58.683 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:58.668 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:58.668 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687dbab00008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4, pid:4559, start at 2024-04-26 19:16:58.67687893 +0800 CST m=+5.657334253 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:58.683 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:58.668 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:58.668 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687dbb6c0015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r6zw4, pid:4631, start at 2024-04-26 19:16:58.760041476 +0800 CST m=+5.684843577 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:18:58.768 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:16:58.764 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:06:58.764 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/processor_resolved_ts_fallback/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/resourcecontrol Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table test.finish_mark not exists for 3-th check, retry later table ddl_manager.finish_mark exists check diff successfully [2024/04/26 19:17:02.218 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 7-th check, retry later ***************** properties ***************** "operationcount"="0" "readproportion"="0" "requestdistribution"="uniform" "scanproportion"="0" "insertproportion"="0" "workload"="core" "mysql.user"="root" "fieldcount"="3" "dotransactions"="false" "mysql.host"="127.0.0.1" "mysql.port"="4000" "updateproportion"="0" "threadcount"="10" "mysql.db"="consistent_replicate_nfs" "readallfields"="true" "recordcount"="50000" ********************************************** [2024/04/26 19:17:02.456 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark7000] wait process cdc.test exit for 1-th time... [2024/04/26 19:17:02.621 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark7000] [ts=449348952606113797] [2024/04/26 19:17:02.621 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark7000] [2024/04/26 19:17:02.621 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348952606113797] [2024/04/26 19:17:02.623 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:02.627 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:02.631 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:02.636 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:02.636 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark12000] table multi_tables_ddl_test.t55 exists table multi_tables_ddl_test.t66 exists table multi_tables_ddl_test.t7 exists table multi_tables_ddl_test.t88 not exists for 1-th check, retry later [2024/04/26 19:17:02.920 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] wait process cdc.test exit for 2-th time... Verifying downstream PD is started... [2024/04/26 19:17:03.234 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:03 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038baa20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348951042424837 MinTableBarrierTs:449348951042424839 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c22eb3a-78c5-4172-93d7-3554d1d56281 {"id":"8c22eb3a-78c5-4172-93d7-3554d1d56281","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2ba8 8c22eb3a-78c5-4172-93d7-3554d1d56281 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348951042424837,"min-table-barrier-ts":449348951042424839,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c22eb3a-78c5-4172-93d7-3554d1d56281/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038baa20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348951042424837 MinTableBarrierTs:449348951042424839 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c22eb3a-78c5-4172-93d7-3554d1d56281 {"id":"8c22eb3a-78c5-4172-93d7-3554d1d56281","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2ba8 8c22eb3a-78c5-4172-93d7-3554d1d56281 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348951042424837,"min-table-barrier-ts":449348951042424839,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c22eb3a-78c5-4172-93d7-3554d1d56281/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0038baa20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348951042424837 MinTableBarrierTs:449348951042424839 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c22eb3a-78c5-4172-93d7-3554d1d56281 {"id":"8c22eb3a-78c5-4172-93d7-3554d1d56281","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2ba8 8c22eb3a-78c5-4172-93d7-3554d1d56281 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348951042424837,"min-table-barrier-ts":449348951042424839,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c22eb3a-78c5-4172-93d7-3554d1d56281/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [2024/04/26 19:17:03.331 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] table multi_tables_ddl_test.t88 exists table multi_tables_ddl_test.finish_mark exists check table exists success wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:03 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/713bbedb-ef7b-44e5-b121-35837e407101 {"id":"713bbedb-ef7b-44e5-b121-35837e407101","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f2d21f7 713bbedb-ef7b-44e5-b121-35837e407101 /tidb/cdc/default/default/upstream/7362133165993337520 {"id":7362133165993337520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/713bbedb-ef7b-44e5-b121-35837e407101 {"id":"713bbedb-ef7b-44e5-b121-35837e407101","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f2d21f7 713bbedb-ef7b-44e5-b121-35837e407101 /tidb/cdc/default/default/upstream/7362133165993337520 {"id":7362133165993337520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/713bbedb-ef7b-44e5-b121-35837e407101 {"id":"713bbedb-ef7b-44e5-b121-35837e407101","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f2d21f7 713bbedb-ef7b-44e5-b121-35837e407101 /tidb/cdc/default/default/upstream/7362133165993337520 {"id":7362133165993337520,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5134.out cli changefeed create '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=true' -c test *************************** 1. row *************************** count(distinct region_id): 13 table test.finish_mark not exists for 4-th check, retry later [Fri Apr 26 19:17:03 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=return(true);github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer/KafkaSinkAsyncSendError=return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.58955897.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Create changefeed successfully! ID: test Info: {"upstream_id":7362133165993337520,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=true","create_time":"2024-04-26T19:17:03.911320646+08:00","start_ts":449348953386254340,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348953386254340,"checkpoint_ts":449348953386254340,"checkpoint_time":"2024-04-26 19:17:03.794"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 4-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + endpoints=http://127.0.0.1:2379 + changefeed_id=test-normal + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-normal -s table mark.finish_mark_3 not exists for 8-th check, retry later [2024/04/26 19:17:04.233 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/lossy_ddl/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:17:02 CST 2024] <<<<<< run test case lossy_ddl success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:04 GMT < Content-Length: 657 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afa192db-9443-4fa9-92fb-a1a457bf06ea {"id":"afa192db-9443-4fa9-92fb-a1a457bf06ea","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f334704 afa192db-9443-4fa9-92fb-a1a457bf06ea /tidb/cdc/default/default/upstream/7362133175660231766 {"id":7362133175660231766,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afa192db-9443-4fa9-92fb-a1a457bf06ea {"id":"afa192db-9443-4fa9-92fb-a1a457bf06ea","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f334704 afa192db-9443-4fa9-92fb-a1a457bf06ea /tidb/cdc/default/default/upstream/7362133175660231766 {"id":7362133175660231766,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/afa192db-9443-4fa9-92fb-a1a457bf06ea {"id":"afa192db-9443-4fa9-92fb-a1a457bf06ea","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f334704 afa192db-9443-4fa9-92fb-a1a457bf06ea /tidb/cdc/default/default/upstream/7362133175660231766 {"id":7362133175660231766,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2779,http://127.0.0.1:2679,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11541.out cli changefeed create --start-ts=449348952216305665 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7362133175660231766,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:17:04.491499325+08:00","start_ts":449348952216305665,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348952216305665,"checkpoint_ts":449348952216305665,"checkpoint_time":"2024-04-26 19:16:59.331"} PASS Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + info='{ "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449348952690524167, "checkpoint_time": "2024-04-26 19:17:01.140", "error": null }' + echo '{ "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449348952690524167, "checkpoint_time": "2024-04-26 19:17:01.140", "error": null }' { "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-normal", "state": "normal", "checkpoint_tso": 449348952690524167, "checkpoint_time": "2024-04-26 19:17:01.140", "error": null } ++ echo '{' '"upstream_id":' 7362133119017085058, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 449348952690524167, '"checkpoint_time":' '"2024-04-26' '19:17:01.140",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362133119017085058, '"namespace":' '"default",' '"id":' '"test-normal",' '"state":' '"normal",' '"checkpoint_tso":' 449348952690524167, '"checkpoint_time":' '"2024-04-26' '19:17:01.140",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-1 + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-1 -s wait process cdc.test exit for 5-th time... [2024/04/26 19:17:04.636 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] coverage: 2.4% of statements in github.com/pingcap/tiflow/... + info='{ "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449348953398312982, "checkpoint_time": "2024-04-26 19:17:03.840", "error": null }' + echo '{ "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449348953398312982, "checkpoint_time": "2024-04-26 19:17:03.840", "error": null }' { "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-error-1", "state": "normal", "checkpoint_tso": 449348953398312982, "checkpoint_time": "2024-04-26 19:17:03.840", "error": null } ++ echo '{' '"upstream_id":' 7362133119017085058, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 449348953398312982, '"checkpoint_time":' '"2024-04-26' '19:17:03.840",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362133119017085058, '"namespace":' '"default",' '"id":' '"test-error-1",' '"state":' '"normal",' '"checkpoint_tso":' 449348953398312982, '"checkpoint_time":' '"2024-04-26' '19:17:03.840",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + endpoints=http://127.0.0.1:2379 + changefeed_id=test-error-2 + expected_state=failed + error_msg=ErrSyncRenameTableFailed + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c test-error-2 -s "is-owner": true, run task successfully + info='{ "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449348952363106305, "checkpoint_time": "2024-04-26 19:16:59.891", "error": { "time": "2024-04-26T19:17:00.985279997+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrHandleDDLFailed", "message": "[CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:17:00.69 +0800 CST, Err:\u003cnil\u003e, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348952572559364. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348952572559364]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' + echo '{ "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449348952363106305, "checkpoint_time": "2024-04-26 19:16:59.891", "error": { "time": "2024-04-26T19:17:00.985279997+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrHandleDDLFailed", "message": "[CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:17:00.69 +0800 CST, Err:\u003cnil\u003e, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348952572559364. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348952572559364]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } }' { "upstream_id": 7362133119017085058, "namespace": "default", "id": "test-error-2", "state": "failed", "checkpoint_tso": 449348952363106305, "checkpoint_time": "2024-04-26 19:16:59.891", "error": { "time": "2024-04-26T19:17:00.985279997+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrHandleDDLFailed", "message": "[CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:17:00.69 +0800 CST, Err:\u003cnil\u003e, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348952572559364. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348952572559364]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '128', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule." } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7362133119017085058, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449348952363106305, '"checkpoint_time":' '"2024-04-26' '19:16:59.891",' '"error":' '{' '"time":' '"2024-04-26T19:17:00.985279997+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrHandleDDLFailed",' '"message":' '"[CDC:ErrHandleDDLFailed]handle' ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:17:00.69 +0800 CST, 'Err:\u003cnil\u003e,' ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348952572559364. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add '`ignore-txn-start-ts=[449348952572559364]`' to the changefeed in the filter configuration.: '[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''128'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7362133119017085058, '"namespace":' '"default",' '"id":' '"test-error-2",' '"state":' '"failed",' '"checkpoint_tso":' 449348952363106305, '"checkpoint_time":' '"2024-04-26' '19:16:59.891",' '"error":' '{' '"time":' '"2024-04-26T19:17:00.985279997+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrHandleDDLFailed",' '"message":' '"[CDC:ErrHandleDDLFailed]handle' ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:17:00.69 +0800 CST, 'Err:\u003cnil\u003e,' ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348952572559364. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add '`ignore-txn-start-ts=[449348952572559364]`' to the changefeed in the filter configuration.: '[CDC:ErrSyncRenameTableFailed]table'\''s' old name is not in filter rule, and its new name in filter rule table id ''\''128'\'',' ddl query: '[rename' table t11 to 't9],' 'it'\''s' an unexpected behavior, if you want to replicate this table, please add its old name to filter 'rule."' '}' '}' + message='[CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:17:00.69 +0800 CST, Err:, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348952572559364. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348952572559364]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table'\''s old name is not in filter rule, and its new name in filter rule table id '\''128'\'', ddl query: [rename table t11 to t9], it'\''s an unexpected behavior, if you want to replicate this table, please add its old name to filter rule.' + [[ ! [CDC:ErrHandleDDLFailed]handle ddl failed, job: ID:140, Type:rename table, State:done, SchemaState:public, SchemaID:102, TableID:128, RowCount:0, ArgLen:0, start time: 2024-04-26 19:17:00.69 +0800 CST, Err:, ErrCount:0, SnapshotVersion:0, LocalMode: false, query: rename table t11 to t9, startTs: 449348952572559364. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[449348952572559364]` to the changefeed in the filter configuration.: [CDC:ErrSyncRenameTableFailed]table's old name is not in filter rule, and its new name in filter rule table id '128', ddl query: [rename table t11 to t9], it's an unexpected behavior, if you want to replicate this table, please add its old name to filter rule. =~ ErrSyncRenameTableFailed ]] check diff successfully wait process cdc.test exit for 6-th time... [2024/04/26 19:17:04.938 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:17:05.129 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark16000] + set +x [2024/04/26 19:17:05.247 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:05.345 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 5-th check, retry later cdc.test: no process found wait process cdc.test exit for 7-th time... process cdc.test already exit [Fri Apr 26 19:17:05 CST 2024] <<<<<< run test case ddl_manager success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x table test.simple exists table test.`simple-dash` not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... owner pid: 4050 owner id 8c22eb3a-78c5-4172-93d7-3554d1d56281 [Fri Apr 26 19:17:06 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/sleep-in-owner-tick=1*sleep(6000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.41604162.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_3 not exists for 9-th check, retry later [2024/04/26 19:16:54.803 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/04/26 19:16:54.803 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7362132182577012573] [2024/04/26 19:16:54.803 +08:00] [INFO] [client.go:606] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/04/26 19:16:54.803 +08:00] [INFO] [tso_client.go:236] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2024/04/26 19:16:54.804 +08:00] [INFO] [tso_dispatcher.go:359] ["[tso] tso dispatcher created"] [dc-location=global] [2024/04/26 19:16:54.804 +08:00] [INFO] [client.go:612] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/04/26 19:16:54.807 +08:00] [INFO] [pd_service_discovery.go:1016] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=] [2024/04/26 19:16:54.807 +08:00] [INFO] [pd_service_discovery.go:498] ["[pd] init cluster id"] [cluster-id=7362132182577012573] [2024/04/26 19:16:54.807 +08:00] [INFO] [client.go:606] ["[pd] changing service mode"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/04/26 19:16:54.808 +08:00] [INFO] [tso_client.go:236] ["[tso] switch dc tso global allocator serving url"] [dc-location=global] [new-url=http://127.0.0.1:2379] [2024/04/26 19:16:54.808 +08:00] [INFO] [tso_dispatcher.go:359] ["[tso] tso dispatcher created"] [dc-location=global] [2024/04/26 19:16:54.808 +08:00] [INFO] [client.go:612] ["[pd] service mode changed"] [old-mode=UNKNOWN_SVC_MODE] [new-mode=PD_SVC_MODE] [2024/04/26 19:16:54.809 +08:00] [INFO] [tikv_driver.go:197] ["using API V1."] [2024/04/26 19:16:54.810 +08:00] [INFO] [main.go:180] ["genLock started"] [2024/04/26 19:16:54.812 +08:00] [INFO] [store_cache.go:510] ["change store resolve state"] [store=1] [addr=127.0.0.1:20162] [from=unresolved] [to=resolved] [liveness-state=reachable] [2024/04/26 19:17:04.825 +08:00] [INFO] [main.go:196] ["genLock done"] [2024/04/26 19:17:04.825 +08:00] [INFO] [tso_dispatcher.go:455] ["[tso] stop fetching the pending tso requests due to context canceled"] [dc-location=global] [2024/04/26 19:17:04.825 +08:00] [INFO] [tso_dispatcher.go:380] ["[tso] exit tso dispatcher"] [dc-location=global] [2024/04/26 19:17:04.825 +08:00] [INFO] [tso_batch_controller.go:158] ["[pd] clear the tso batch controller"] [max-batch-size=10000] [best-batch-size=1] [collected-request-count=0] [pending-request-count=0] [2024/04/26 19:17:04.826 +08:00] [INFO] [tso_dispatcher.go:268] ["exit tso dispatcher loop"] [2024/04/26 19:17:04.826 +08:00] [INFO] [tso_dispatcher.go:214] ["exit tso requests cancel loop"] [2024/04/26 19:17:04.826 +08:00] [INFO] [resource_manager_client.go:295] ["[resource manager] exit resource token dispatcher"] [2024/04/26 19:17:04.826 +08:00] [INFO] [pd_service_discovery.go:550] ["[pd] exit member loop due to context canceled"] [2024/04/26 19:17:04.826 +08:00] [INFO] [tso_client.go:140] ["closing tso client"] [2024/04/26 19:17:04.826 +08:00] [INFO] [tso_client.go:145] ["close tso client"] [2024/04/26 19:17:04.826 +08:00] [INFO] [tso_batch_controller.go:158] ["[pd] clear the tso batch controller"] [max-batch-size=10000] [best-batch-size=1] [collected-request-count=0] [pending-request-count=0] [2024/04/26 19:17:04.826 +08:00] [INFO] [tso_client.go:155] ["tso client is closed"] [2024/04/26 19:17:04.826 +08:00] [INFO] [pd_service_discovery.go:637] ["[pd] close pd service discovery client"] =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_csv_update/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:17:05 CST 2024] <<<<<< run test case storage_csv_update success! >>>>>> [2024/04/26 19:17:06.246 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > [2024/04/26 19:17:06.648 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:06 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/52ec549e-9115-4d35-a597-bd4464f89384 {"id":"52ec549e-9115-4d35-a597-bd4464f89384","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a82f3 52ec549e-9115-4d35-a597-bd4464f89384 /tidb/cdc/default/default/upstream/7362133189865359228 {"id":7362133189865359228,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/52ec549e-9115-4d35-a597-bd4464f89384 {"id":"52ec549e-9115-4d35-a597-bd4464f89384","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a82f3 52ec549e-9115-4d35-a597-bd4464f89384 /tidb/cdc/default/default/upstream/7362133189865359228 {"id":7362133189865359228,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/52ec549e-9115-4d35-a597-bd4464f89384 {"id":"52ec549e-9115-4d35-a597-bd4464f89384","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a82f3 52ec549e-9115-4d35-a597-bd4464f89384 /tidb/cdc/default/default/upstream/7362133189865359228 {"id":7362133189865359228,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.cli.5947.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8301 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:17:06 CST 2024] <<<<<< run test case multi_tables_ddl success! >>>>>> Exiting on signal: INTERRUPT Create changefeed successfully! ID: e4e3593c-c360-4726-87c2-43e406e2e649 Info: {"upstream_id":7362133189865359228,"namespace":"default","id":"e4e3593c-c360-4726-87c2-43e406e2e649","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:17:06.859994611+08:00","start_ts":449348954152763397,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348954152763397,"checkpoint_ts":449348954152763397,"checkpoint_time":"2024-04-26 19:17:06.718"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:17:06.951 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:17:07.261 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:07.357 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] table test.finish_mark not exists for 6-th check, retry later [2024/04/26 19:17:07.729 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark17000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.`simple-dash` not exists for 2-th check, retry later table mark.finish_mark_3 not exists for 10-th check, retry later [2024/04/26 19:17:08.318 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] + set +x [2024/04/26 19:17:08.721 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table region_merge.t1 exists check diff failed 1-th time, retry later [2024/04/26 19:17:09.022 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:17:09.330 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:09 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00303e630 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348954450034698 MinTableBarrierTs:449348954712178701 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c22eb3a-78c5-4172-93d7-3554d1d56281 {"id":"8c22eb3a-78c5-4172-93d7-3554d1d56281","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/efce9117-ad73-4e87-9822-b39a006fa2c0 {"id":"efce9117-ad73-4e87-9822-b39a006fa2c0","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2ba8 8c22eb3a-78c5-4172-93d7-3554d1d56281 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2bdd efce9117-ad73-4e87-9822-b39a006fa2c0 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348954450034698,"min-table-barrier-ts":449348954712178701,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c22eb3a-78c5-4172-93d7-3554d1d56281/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/efce9117-ad73-4e87-9822-b39a006fa2c0/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00303e630 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348954450034698 MinTableBarrierTs:449348954712178701 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c22eb3a-78c5-4172-93d7-3554d1d56281 {"id":"8c22eb3a-78c5-4172-93d7-3554d1d56281","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/efce9117-ad73-4e87-9822-b39a006fa2c0 {"id":"efce9117-ad73-4e87-9822-b39a006fa2c0","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2ba8 8c22eb3a-78c5-4172-93d7-3554d1d56281 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2bdd efce9117-ad73-4e87-9822-b39a006fa2c0 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348954450034698,"min-table-barrier-ts":449348954712178701,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c22eb3a-78c5-4172-93d7-3554d1d56281/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/efce9117-ad73-4e87-9822-b39a006fa2c0/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00303e630 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348954450034698 MinTableBarrierTs:449348954712178701 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c22eb3a-78c5-4172-93d7-3554d1d56281 {"id":"8c22eb3a-78c5-4172-93d7-3554d1d56281","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/efce9117-ad73-4e87-9822-b39a006fa2c0 {"id":"efce9117-ad73-4e87-9822-b39a006fa2c0","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2ba8 8c22eb3a-78c5-4172-93d7-3554d1d56281 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2bdd efce9117-ad73-4e87-9822-b39a006fa2c0 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348954450034698,"min-table-barrier-ts":449348954712178701,"admin-job-type":0} /tidb/cdc/default/default/task/position/8c22eb3a-78c5-4172-93d7-3554d1d56281/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/efce9117-ad73-4e87-9822-b39a006fa2c0/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "8c22eb3a-78c5-4172-93d7-3554d1d56281" | grep id \033[0;36m<<< Run all test success >>>\033[0m [2024/04/26 19:17:09.427 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark8000] [2024/04/26 19:17:09.521 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark8000] [ts=449348954414907396] [2024/04/26 19:17:09.521 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark8000] [2024/04/26 19:17:09.521 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348954414907396] [2024/04/26 19:17:09.525 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:09.529 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:09.534 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:09.539 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:09.539 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark13000] [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^1$' 1 run task successfully table processor_resolved_ts_fallback.t1 exists [Fri Apr 26 19:17:09 CST 2024] <<<<<< START cdc server in processor_resolved_ts_fallback case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.processor_resolved_ts_fallback.60306032.out server --log-file /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/processor_resolved_ts_fallback/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Pipeline] // cache [Pipeline] } [Pipeline] // dir table mark.finish_mark_3 not exists for 11-th check, retry later [Pipeline] } table test.finish_mark not exists for 7-th check, retry later [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // stage [Pipeline] } table test.`simple-dash` exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348954942865413, "checkpoint_time": "2024-04-26 19:17:09.732", "error": null }' + echo '{ "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348954942865413, "checkpoint_time": "2024-04-26 19:17:09.732", "error": null }' { "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348954942865413, "checkpoint_time": "2024-04-26 19:17:09.732", "error": null } ++ echo '{' '"upstream_id":' 7362133175660231766, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449348954942865413, '"checkpoint_time":' '"2024-04-26' '19:17:09.732",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362133175660231766, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449348954942865413, '"checkpoint_time":' '"2024-04-26' '19:17:09.732",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379 [Pipeline] // container [2024/04/26 19:17:10.334 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] [Pipeline] } + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5176.out cli changefeed pause -c test [Pipeline] // withEnv changefeed count 1 check pass, pd_addr: http://127.0.0.1:2679 changefeed count 1 check pass, pd_addr: http://127.0.0.1:2779 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } "cluster-id": "default" "id": "efce9117-ad73-4e87-9822-b39a006fa2c0", "cluster-id": "default" run task successfully [2024/04/26 19:17:10.737 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] [Pipeline] // node changefeed count 1 check pass, pd_addr: http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 [Pipeline] } [2024/04/26 19:17:11.033 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } table test.t2 not exists for 1-th check, retry later [Pipeline] // stage [Pipeline] } check diff successfully Error: [CDC:ErrChangefeedUpdateRefused]changefeed update error: can only update changefeed config when it is stopped or failed update changefeed config should fail when changefeed is running, got Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11818.out cli changefeed --changefeed-id custom-changefeed-name pause [2024/04/26 19:17:11.423 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:11.539 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] wait process cdc.test exit for 1-th time... [2024/04/26 19:17:11.749 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark18000] table test.finish_mark not exists for 8-th check, retry later capture_id: efce9117-ad73-4e87-9822-b39a006fa2c0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) INSERT - Takes(s): 10.0, Count: 20751, OPS: 2076.4, Avg(us): 4799, Min(us): 735, Max(us): 92988, 95th(us): 8000, 99th(us): 87000 wait process cdc.test exit for 2-th time... [2024/04/26 19:17:12.347 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] table mark.finish_mark_3 not exists for 12-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5205.out cli changefeed update -c test '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=false' --no-confirm ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Diff of changefeed config: {Type:update Path:[SinkURI] From:mysql://root@127.0.0.1:3306/?batch-dml-enable=true To:mysql://root@127.0.0.1:3306/?batch-dml-enable=false} table test.t2 not exists for 2-th check, retry later PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:12 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/e4e3593c-c360-4726-87c2-43e406e2e649 {UpstreamID:7362133189865359228 Namespace:default ID:e4e3593c-c360-4726-87c2-43e406e2e649 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:17:06.859994611 +0800 CST StartTs:449348954152763397 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031222d0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348954178977798} {CheckpointTs:449348955555233798 MinTableBarrierTs:449348955555233798 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d8bb34f-9f9a-4a74-9766-aba7305cf543 {"id":"2d8bb34f-9f9a-4a74-9766-aba7305cf543","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/52ec549e-9115-4d35-a597-bd4464f89384 {"id":"52ec549e-9115-4d35-a597-bd4464f89384","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a82f3 52ec549e-9115-4d35-a597-bd4464f89384 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a834a 2d8bb34f-9f9a-4a74-9766-aba7305cf543 /tidb/cdc/default/default/changefeed/info/e4e3593c-c360-4726-87c2-43e406e2e649 {"upstream-id":7362133189865359228,"namespace":"default","changefeed-id":"e4e3593c-c360-4726-87c2-43e406e2e649","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:17:06.859994611+08:00","start-ts":449348954152763397,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348954178977798} /tidb/cdc/default/default/changefeed/status/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":449348955555233798,"min-table-barrier-ts":449348955555233798,"admin-job-type":0} /tidb/cdc/default/default/task/position/2d8bb34f-9f9a-4a74-9766-aba7305cf543/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/52ec549e-9115-4d35-a597-bd4464f89384/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133189865359228 {"id":7362133189865359228,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/e4e3593c-c360-4726-87c2-43e406e2e649 {UpstreamID:7362133189865359228 Namespace:default ID:e4e3593c-c360-4726-87c2-43e406e2e649 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:17:06.859994611 +0800 CST StartTs:449348954152763397 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031222d0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348954178977798} {CheckpointTs:449348955555233798 MinTableBarrierTs:449348955555233798 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d8bb34f-9f9a-4a74-9766-aba7305cf543 {"id":"2d8bb34f-9f9a-4a74-9766-aba7305cf543","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/52ec549e-9115-4d35-a597-bd4464f89384 {"id":"52ec549e-9115-4d35-a597-bd4464f89384","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a82f3 52ec549e-9115-4d35-a597-bd4464f89384 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a834a 2d8bb34f-9f9a-4a74-9766-aba7305cf543 /tidb/cdc/default/default/changefeed/info/e4e3593c-c360-4726-87c2-43e406e2e649 {"upstream-id":7362133189865359228,"namespace":"default","changefeed-id":"e4e3593c-c360-4726-87c2-43e406e2e649","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:17:06.859994611+08:00","start-ts":449348954152763397,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348954178977798} /tidb/cdc/default/default/changefeed/status/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":449348955555233798,"min-table-barrier-ts":449348955555233798,"admin-job-type":0} /tidb/cdc/default/default/task/position/2d8bb34f-9f9a-4a74-9766-aba7305cf543/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/52ec549e-9115-4d35-a597-bd4464f89384/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133189865359228 {"id":7362133189865359228,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/e4e3593c-c360-4726-87c2-43e406e2e649 {UpstreamID:7362133189865359228 Namespace:default ID:e4e3593c-c360-4726-87c2-43e406e2e649 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:17:06.859994611 +0800 CST StartTs:449348954152763397 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0031222d0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348954178977798} {CheckpointTs:449348955555233798 MinTableBarrierTs:449348955555233798 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d8bb34f-9f9a-4a74-9766-aba7305cf543 {"id":"2d8bb34f-9f9a-4a74-9766-aba7305cf543","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/52ec549e-9115-4d35-a597-bd4464f89384 {"id":"52ec549e-9115-4d35-a597-bd4464f89384","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a82f3 52ec549e-9115-4d35-a597-bd4464f89384 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4a834a 2d8bb34f-9f9a-4a74-9766-aba7305cf543 /tidb/cdc/default/default/changefeed/info/e4e3593c-c360-4726-87c2-43e406e2e649 {"upstream-id":7362133189865359228,"namespace":"default","changefeed-id":"e4e3593c-c360-4726-87c2-43e406e2e649","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:17:06.859994611+08:00","start-ts":449348954152763397,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348954178977798} /tidb/cdc/default/default/changefeed/status/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":449348955555233798,"min-table-barrier-ts":449348955555233798,"admin-job-type":0} /tidb/cdc/default/default/task/position/2d8bb34f-9f9a-4a74-9766-aba7305cf543/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/52ec549e-9115-4d35-a597-bd4464f89384/e4e3593c-c360-4726-87c2-43e406e2e649 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133189865359228 {"id":7362133189865359228,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x [2024/04/26 19:17:12.750 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] Update changefeed config successfully! ID: test Info: {"upstream_id":7362133165993337520,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=false","create_time":"2024-04-26T19:17:03.911320646+08:00","start_ts":449348953386254340,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348955037761540,"checkpoint_ts":449348954198900762,"checkpoint_time":"2024-04-26 19:17:06.894"} PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:17:13 CST 2024] <<<<<< run test case region_merge success! >>>>>> [2024/04/26 19:17:13.045 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:17:13.436 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:13.555 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] table test.finish_mark not exists for 9-th check, retry later table processor_resolved_ts_fallback.t2 exists table processor_resolved_ts_fallback.t3 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x table mark.finish_mark_3 not exists for 13-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5231.out cli changefeed resume -c test [2024/04/26 19:17:14.419 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] table test.t2 not exists for 3-th check, retry later [2024/04/26 19:17:14.764 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687e88fc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:9816, start at 2024-04-26 19:17:11.892640317 +0800 CST m=+5.286595947 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:11.900 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:11.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:11.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687e88fc0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:9816, start at 2024-04-26 19:17:11.892640317 +0800 CST m=+5.286595947 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:11.900 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:11.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:11.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687e8a78000b Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-95t2d, pid:9891, start at 2024-04-26 19:17:11.978537668 +0800 CST m=+5.322164651 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:11.989 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:11.966 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:11.966 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/resourcecontrol/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash-proxy.toml"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/resourcecontrol/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/04/26 19:17:15.058 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark9000] [2024/04/26 19:17:15.222 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark9000] [ts=449348956223701000] [2024/04/26 19:17:15.222 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark9000] [2024/04/26 19:17:15.222 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348956223701000] [2024/04/26 19:17:15.225 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:15.318 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:15.325 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:15.418 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:15.418 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark14000] [2024/04/26 19:17:15.419 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark19000] nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table test.finish_mark not exists for 10-th check, retry later [2024/04/26 19:17:15.518 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:15.620 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] table processor_resolved_ts_fallback.t3 exists cdc cli processor list --server http://127.0.0.1:8301 |jq '.|length'|grep -E '^2$' 2 run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 14-th check, retry later + set +x [2024/04/26 19:17:16.432 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] \033[0;36m<<< Run all test success >>>\033[0m killing cdc1: 5900 cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) table processor_delay.t1 exists table processor_delay.t2 exists table processor_delay.t3 exists table processor_delay.t4 exists table processor_delay.t5 exists table processor_delay.t6 exists table processor_delay.t7 exists table processor_delay.t8 exists table processor_delay.t9 not exists for 1-th check, retry later [Pipeline] // cache [2024/04/26 19:17:16.776 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] [Pipeline] } Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 1-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11219.out cli tso query --pd=http://127.0.0.1:2379 table test.t2 not exists for 4-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [2024/04/26 19:17:17.419 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=stopped + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449348955467153413, "checkpoint_time": "2024-04-26 19:17:11.732", "error": null }' + echo '{ "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449348955467153413, "checkpoint_time": "2024-04-26 19:17:11.732", "error": null }' { "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "stopped", "checkpoint_tso": 449348955467153413, "checkpoint_time": "2024-04-26 19:17:11.732", "error": null } ++ echo '{' '"upstream_id":' 7362133175660231766, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449348955467153413, '"checkpoint_time":' '"2024-04-26' '19:17:11.732",' '"error":' null '}' ++ jq -r .state + state=stopped + [[ ! stopped == \s\t\o\p\p\e\d ]] ++ echo '{' '"upstream_id":' 7362133175660231766, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"stopped",' '"checkpoint_tso":' 449348955467153413, '"checkpoint_time":' '"2024-04-26' '19:17:11.732",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11876.out cli changefeed update --pd=http://127.0.0.1:2379,http://127.0.0.1:2679,http://127.0.0.1:2779 --config=/tmp/tidb_cdc_test/cli/changefeed.toml --no-confirm --changefeed-id custom-changefeed-name table test.finish_mark not exists for 11-th check, retry later [2024/04/26 19:17:17.618 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:17.720 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ee0ac0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:6481, start at 2024-04-26 19:17:17.499395738 +0800 CST m=+21.825302359 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:17.510 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:17.483 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:17.483 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // dir Diff of changefeed config: {Type:update Path:[Config CaseSensitive] From:false To:true} {Type:update Path:[Config Scheduler EnableTableAcrossNodes] From:false To:true} Update changefeed config successfully! ID: custom-changefeed-name Info: {"upstream_id":7362133175660231766,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:17:04.491499325+08:00","start_ts":449348952216305665,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":true,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348955467153414,"checkpoint_ts":449348955467153413,"checkpoint_time":"2024-04-26 19:17:11.732"} PASS coverage: 2.9% of statements in github.com/pingcap/tiflow/... table mark.finish_mark_3 not exists for 15-th check, retry later [2024/04/26 19:17:18.063 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20001] [2024/04/26 19:17:18.064 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20002] [2024/04/26 19:17:18.118 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20003] [2024/04/26 19:17:18.119 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20004] [Pipeline] } [2024/04/26 19:17:18.446 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] table processor_delay.t9 exists table processor_delay.t10 not exists for 1-th check, retry later cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5272.out cli changefeed pause -c test PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... + set +x + tso='449348956906061825 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348956906061825 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:17:18 CST 2024] <<<<<< START cdc server in resourcecontrol case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.1126111263.out server --log-file /tmp/tidb_cdc_test/resourcecontrol/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/resourcecontrol/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:17:18.835 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] run task failed 2-th time, retry later table test.t2 exists check diff successfully nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [Pipeline] // withCredentials + set +x wait process cdc.test exit for 1-th time... [2024/04/26 19:17:19.432 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [2024/04/26 19:17:19.630 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11920.out cli changefeed --changefeed-id custom-changefeed-name resume [2024/04/26 19:17:14.888 +08:00] [WARN] [diff.go:182] ["table struct is not equal"] [reason="column num not equal, one is 34 another is 33"] [2024/04/26 19:17:19.733 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] [Pipeline] } wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 12-th check, retry later table mark.finish_mark_3 not exists for 16-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5301.out cli changefeed update -c test '--sink-uri=mysql://root@127.0.0.1:3306/?batch-dml-enable=true' --no-confirm PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687ee0ac0012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:6481, start at 2024-04-26 19:17:17.499395738 +0800 CST m=+21.825302359 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:17.510 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:17.483 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:17.483 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c687e0e6c0011 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tss7n, pid:6557, start at 2024-04-26 19:17:04.047029388 +0800 CST m=+8.299241878 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:04.055 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:04.027 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:04.027 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_rocks/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_rocks/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/04/26 19:17:20.459 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark10000] [Pipeline] // timeout wait process cdc.test exit for 3-th time... [2024/04/26 19:17:20.538 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark10000] [ts=449348957541498885] [2024/04/26 19:17:20.538 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark10000] [2024/04/26 19:17:20.538 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348957541498885] [2024/04/26 19:17:20.540 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:20.620 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:20.625 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:20.629 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:20.629 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark15000] [2024/04/26 19:17:20.629 +08:00] [INFO] [case.go:539] ["upstream create table executed"] [tblName=finishmark20000] [2024/04/26 19:17:20.629 +08:00] [INFO] [case.go:556] ["one upstream workload finished"] [round=20004] table processor_delay.t10 exists table processor_delay.t11 not exists for 1-th check, retry later Diff of changefeed config: {Type:update Path:[SinkURI] From:mysql://root@127.0.0.1:3306/?batch-dml-enable=false To:mysql://root@127.0.0.1:3306/?batch-dml-enable=true} Update changefeed config successfully! ID: test Info: {"upstream_id":7362133165993337520,"namespace":"default","id":"test","sink_uri":"mysql://root@127.0.0.1:3306/?batch-dml-enable=true","create_time":"2024-04-26T19:17:03.911320646+08:00","start_ts":449348953386254340,"admin_job_type":1,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"delete_only_output_handle_key_columns":null,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"stopped","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348957134651406,"checkpoint_ts":449348954473889805,"checkpoint_time":"2024-04-26 19:17:07.943"} PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... [Pipeline] } [2024/04/26 19:17:20.848 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] [Pipeline] // stage [Pipeline] } empty select id, val from test.availability1 where id=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:17:21 CST 2024] <<<<<< run test case resolve_lock success! >>>>>> [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [2024/04/26 19:17:21.443 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [Pipeline] // node [Pipeline] } + set +x [Pipeline] // podTemplate [Pipeline] } [2024/04/26 19:17:21.644 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:21.749 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage table test.finish_mark not exists for 13-th check, retry later [Pipeline] } table mark.finish_mark_3 not exists for 17-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.batch_update_to_no_batch.cli.5325.out cli changefeed resume -c test + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:21 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce6b3260-40cd-4c95-9da0-8345ecd6430a {"id":"ce6b3260-40cd-4c95-9da0-8345ecd6430a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f7998f6 ce6b3260-40cd-4c95-9da0-8345ecd6430a /tidb/cdc/default/default/upstream/7362133241805386648 {"id":7362133241805386648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce6b3260-40cd-4c95-9da0-8345ecd6430a {"id":"ce6b3260-40cd-4c95-9da0-8345ecd6430a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f7998f6 ce6b3260-40cd-4c95-9da0-8345ecd6430a /tidb/cdc/default/default/upstream/7362133241805386648 {"id":7362133241805386648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ce6b3260-40cd-4c95-9da0-8345ecd6430a {"id":"ce6b3260-40cd-4c95-9da0-8345ecd6430a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f7998f6 ce6b3260-40cd-4c95-9da0-8345ecd6430a /tidb/cdc/default/default/upstream/7362133241805386648 {"id":7362133241805386648,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.resourcecontrol.cli.11305.out cli changefeed create --start-ts=449348956906061825 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: acc3ae3f-c209-4606-898c-3f5eb0416fc3 Info: {"upstream_id":7362133241805386648,"namespace":"default","id":"acc3ae3f-c209-4606-898c-3f5eb0416fc3","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:17:22.127561658+08:00","start_ts":449348956906061825,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348956906061825,"checkpoint_ts":449348956906061825,"checkpoint_time":"2024-04-26 19:17:17.221"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Fri Apr 26 19:17:22 CST 2024] <<<<<< START cdc server in multi_rocks case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.1940119403.out server --log-file /tmp/tidb_cdc_test/multi_rocks/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_rocks/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table processor_delay.t11 exists table processor_delay.t12 not exists for 1-th check, retry later [2024/04/26 19:17:22.630 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] PASS [2024/04/26 19:17:22.918 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] coverage: 2.1% of statements in github.com/pingcap/tiflow/... cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' Error: Get "http://127.0.0.1:8301/api/v2/captures": dial tcp 127.0.0.1:8301: connect: connection refused run task failed 3-th time, retry later [2024/04/26 19:17:23.455 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] + set +x nonempty select id, val from test.availability2 where id=1 and val=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [2024/04/26 19:17:23.718 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark11000] [2024/04/26 19:17:23.761 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] table test.finish_mark not exists for 14-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_source/run.sh using Sink-Type: mysql... <<================= [2024/04/26 19:17:23.820 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark11000] [ts=449348958459002885] [2024/04/26 19:17:23.820 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark11000] [2024/04/26 19:17:23.820 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348958459002885] [2024/04/26 19:17:23.824 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:23.827 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:23.831 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:23.836 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:23.836 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] + set +x INSERT - Takes(s): 20.1, Count: 30693, OPS: 1529.0, Avg(us): 6494, Min(us): 735, Max(us): 94921, 95th(us): 76000, 99th(us): 88000 table mark.finish_mark_3 not exists for 18-th check, retry later table processor_delay.t12 exists table processor_delay.t13 not exists for 1-th check, retry later The 1 times to try to start tidb cluster... table batch_update_to_no_batch.v1 exists table batch_update_to_no_batch.recover_and_insert exists table batch_update_to_no_batch.finish_mark not exists for 1-th check, retry later + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348958612619271, "checkpoint_time": "2024-04-26 19:17:23.731", "error": null }' + echo '{ "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348958612619271, "checkpoint_time": "2024-04-26 19:17:23.731", "error": null }' { "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348958612619271, "checkpoint_time": "2024-04-26 19:17:23.731", "error": null } ++ echo '{' '"upstream_id":' 7362133175660231766, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449348958612619271, '"checkpoint_time":' '"2024-04-26' '19:17:23.731",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362133175660231766, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449348958612619271, '"checkpoint_time":' '"2024-04-26' '19:17:23.731",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.11996.out cli changefeed --changefeed-id custom-changefeed-name remove [2024/04/26 19:17:24.643 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] [2024/04/26 19:17:24.930 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] Changefeed remove successfully. ID: custom-changefeed-name CheckpointTs: 449348958875025418 SinkURI: mysql://normal:xxxxx@127.0.0.1:3306/ PASS =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/common_1/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... nonempty select id, val from test.availability2 where id=1 and val=22 run task successfully empty select id, val from test.availability2 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:25 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c2422f9-2f2f-4171-bead-765ffae2fb58 {"id":"8c2422f9-2f2f-4171-bead-765ffae2fb58","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4b2905 8c2422f9-2f2f-4171-bead-765ffae2fb58 /tidb/cdc/default/default/upstream/7362133201857380590 {"id":7362133201857380590,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c2422f9-2f2f-4171-bead-765ffae2fb58 {"id":"8c2422f9-2f2f-4171-bead-765ffae2fb58","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4b2905 8c2422f9-2f2f-4171-bead-765ffae2fb58 /tidb/cdc/default/default/upstream/7362133201857380590 {"id":7362133201857380590,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8c2422f9-2f2f-4171-bead-765ffae2fb58 {"id":"8c2422f9-2f2f-4171-bead-765ffae2fb58","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1f4b2905 8c2422f9-2f2f-4171-bead-765ffae2fb58 /tidb/cdc/default/default/upstream/7362133201857380590 {"id":7362133201857380590,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_rocks.cli.20169.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ coverage: 2.1% of statements in github.com/pingcap/tiflow/... table test.finish_mark not exists for 15-th check, retry later Create changefeed successfully! ID: c64ee369-6b80-4007-baee-829045dfb23d Info: {"upstream_id":7362133201857380590,"namespace":"default","id":"c64ee369-6b80-4007-baee-829045dfb23d","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:17:25.579211943+08:00","start_ts":449348959058526215,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348959058526215,"checkpoint_ts":449348959058526215,"checkpoint_time":"2024-04-26 19:17:25.432"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [2024/04/26 19:17:25.527 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [2024/04/26 19:17:25.835 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] table mark.finish_mark_3 not exists for 19-th check, retry later table batch_update_to_no_batch.finish_mark not exists for 2-th check, retry later [2024/04/26 19:17:26.655 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] table resourcecontrol.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later check data successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:17:24 CST 2024] <<<<<< run test case ddl_puller_lag success! >>>>>> [2024/04/26 19:17:26.943 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark12000] [2024/04/26 19:17:27.023 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark12000] [ts=449348959376506885] [2024/04/26 19:17:27.023 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark12000] [2024/04/26 19:17:27.023 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348959376506885] [2024/04/26 19:17:27.025 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:27.028 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:27.035 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:27.039 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:27.040 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark16000] + set +x table processor_delay.t13 exists table processor_delay.t14 not exists for 1-th check, retry later + set +x wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/common_1 Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... empty select id, val from test.availability2 where id=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ***************** properties ***************** "requestdistribution"="uniform" "operationcount"="0" "mysql.host"="127.0.0.1" "updateproportion"="0" "insertproportion"="0" "dotransactions"="false" "readallfields"="true" "recordcount"="1000" "mysql.port"="4000" "scanproportion"="0" "threadcount"="2" "table"="a1" "mysql.user"="root" "workload"="core" "mysql.db"="multi_rocks" "readproportion"="0" ********************************************** [2024/04/26 19:17:27.540 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] table test.finish_mark not exists for 16-th check, retry later Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [2024/04/26 19:17:27.847 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:17:27 CST 2024] <<<<<< run test case resourcecontrol success! >>>>>> Run finished, takes 654.570649ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1572.7, Avg(us): 1274, Min(us): 919, Max(us): 18658, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "mysql.db"="multi_rocks" "threadcount"="2" "updateproportion"="0" "readproportion"="0" "mysql.host"="127.0.0.1" "dotransactions"="false" "operationcount"="0" "mysql.port"="4000" "scanproportion"="0" "requestdistribution"="uniform" "insertproportion"="0" "mysql.user"="root" "workload"="core" "table"="a2" "readallfields"="true" "recordcount"="1000" ********************************************** table mark.finish_mark_3 not exists for 20-th check, retry later table batch_update_to_no_batch.finish_mark not exists for 3-th check, retry later [2024/04/26 19:17:28.667 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] Run finished, takes 696.306366ms INSERT - Takes(s): 0.7, Count: 1000, OPS: 1483.4, Avg(us): 1359, Min(us): 959, Max(us): 22141, 95th(us): 2000, 99th(us): 2000 ***************** properties ***************** "requestdistribution"="uniform" "workload"="core" "mysql.host"="127.0.0.1" "readallfields"="true" "mysql.db"="multi_rocks" "dotransactions"="false" "table"="a3" "mysql.port"="4000" "recordcount"="1000" "threadcount"="2" "insertproportion"="0" "readproportion"="0" "operationcount"="0" "mysql.user"="root" "scanproportion"="0" "updateproportion"="0" ********************************************** table processor_delay.t14 exists table processor_delay.t15 not exists for 1-th check, retry later [2024/04/26 19:17:29.040 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Run finished, takes 658.51851ms INSERT - Takes(s): 0.6, Count: 1000, OPS: 1569.1, Avg(us): 1279, Min(us): 886, Max(us): 21235, 95th(us): 2000, 99th(us): 3000 ***************** properties ***************** "scanproportion"="0" "dotransactions"="false" "operationcount"="0" "recordcount"="1000" "readproportion"="0" "mysql.db"="multi_rocks" "updateproportion"="0" "workload"="core" "mysql.port"="4000" "mysql.host"="127.0.0.1" "table"="a4" "threadcount"="2" "readallfields"="true" "insertproportion"="0" "mysql.user"="root" "requestdistribution"="uniform" ********************************************** cdc cli capture list --server http://127.0.0.1:8302 |jq '.|length'|grep -E '^1$' 1 run task successfully check diff failed 1-th time, retry later [2024/04/26 19:17:29.618 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [2024/04/26 19:17:29.860 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] nonempty select id, val from test.availability3 where id=1 and val=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table test.finish_mark not exists for 17-th check, retry later table mark.finish_mark_3 not exists for 21-th check, retry later [2024/04/26 19:17:30.718 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] Run finished, takes 1.296837178s INSERT - Takes(s): 1.3, Count: 1000, OPS: 786.9, Avg(us): 2555, Min(us): 907, Max(us): 77412, 95th(us): 3000, 99th(us): 67000 table processor_delay.t15 exists table processor_delay.t16 not exists for 1-th check, retry later [2024/04/26 19:17:31.052 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] table batch_update_to_no_batch.finish_mark not exists for 4-th check, retry later ***************** properties ***************** "operationcount"="0" "updateproportion"="0" "workload"="core" "scanproportion"="0" "readproportion"="0" "threadcount"="2" "readallfields"="true" "recordcount"="1000" "insertproportion"="0" "table"="a5" "mysql.db"="multi_rocks" "dotransactions"="false" "mysql.port"="4000" "requestdistribution"="uniform" "mysql.user"="root" "mysql.host"="127.0.0.1" ********************************************** Starting Upstream TiDB... [2024/04/26 19:17:31.630 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] check diff successfully nonempty select id, val from test.availability3 where id=1 and val=22 run task successfully empty select id, val from test.availability3 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Run finished, takes 662.003266ms INSERT - Takes(s): 0.6, Count: 999, OPS: 1562.3, Avg(us): 1259, Min(us): 908, Max(us): 22417, 95th(us): 2000, 99th(us): 3000 Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:17:31.873 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark13000] wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 18-th check, retry later table mark.finish_mark_3 not exists for 22-th check, retry later [2024/04/26 19:17:31.887 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark13000] [ts=449348960294010884] [2024/04/26 19:17:31.887 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark13000] [2024/04/26 19:17:31.887 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348960294010884] [2024/04/26 19:17:31.890 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:31.892 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:31.897 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:31.901 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:31.902 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark17000] changefeed count 0 check pass, pd_addr: http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.12069.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --tz=Asia/Shanghai -c=custom-changefeed-name [WARN] --tz is deprecated in changefeed settings. Create changefeed successfully! ID: custom-changefeed-name Info: {"upstream_id":7362133175660231766,"namespace":"default","id":"custom-changefeed-name","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:17:31.796395566+08:00","start_ts":449348960683819012,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348960683819012,"checkpoint_ts":449348960683819012,"checkpoint_time":"2024-04-26 19:17:31.632"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table multi_rocks.finish_mark not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... INSERT - Takes(s): 30.0, Count: 40588, OPS: 1353.2, Avg(us): 7370, Min(us): 735, Max(us): 363934, 95th(us): 81000, 99th(us): 88000 table processor_delay.t16 exists table processor_delay.t17 not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:17:32 CST 2024] <<<<<< run test case processor_resolved_ts_fallback success! >>>>>> table batch_update_to_no_batch.finish_mark not exists for 5-th check, retry later [2024/04/26 19:17:32.735 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] [2024/04/26 19:17:33.136 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] + set +x empty select id, val from test.availability3 where id=1 run task successfully test_gap_between_watch_capture pass ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:17:33.643 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark14000] [2024/04/26 19:17:33.657 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark14000] [ts=449348961211514885] [2024/04/26 19:17:33.657 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark14000] [2024/04/26 19:17:33.657 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348961211514885] [2024/04/26 19:17:33.659 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:33.661 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:33.665 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:33.668 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:33.668 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] table test.finish_mark not exists for 19-th check, retry later [2024/04/26 19:17:33.918 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] wait process cdc.test exit for 1-th time... table mark.finish_mark_3 not exists for 23-th check, retry later wait process cdc.test exit for 2-th time... table multi_rocks.finish_mark not exists for 2-th check, retry later table processor_delay.t17 exists table processor_delay.t18 not exists for 1-th check, retry later table batch_update_to_no_batch.finish_mark exists check diff successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case delete_owner_key [Fri Apr 26 19:17:35 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.45574559.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server1 --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info [2024/04/26 19:17:34.748 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:17:35.148 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 20-th check, retry later wait process cdc.test exit for 2-th time... [2024/04/26 19:17:35.935 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 24-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:17:36 CST 2024] <<<<<< run test case batch_update_to_no_batch success! >>>>>> table multi_rocks.finish_mark not exists for 3-th check, retry later table processor_delay.t18 exists + endpoints=http://127.0.0.1:2379 + changefeed_id=custom-changefeed-name + expected_state=normal + error_msg=null + tls_dir= + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c custom-changefeed-name -s + info='{ "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348961771454469, "checkpoint_time": "2024-04-26 19:17:35.781", "error": null }' + echo '{ "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348961771454469, "checkpoint_time": "2024-04-26 19:17:35.781", "error": null }' { "upstream_id": 7362133175660231766, "namespace": "default", "id": "custom-changefeed-name", "state": "normal", "checkpoint_tso": 449348961771454469, "checkpoint_time": "2024-04-26 19:17:35.781", "error": null } ++ echo '{' '"upstream_id":' 7362133175660231766, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449348961771454469, '"checkpoint_time":' '"2024-04-26' '19:17:35.781",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362133175660231766, '"namespace":' '"default",' '"id":' '"custom-changefeed-name",' '"state":' '"normal",' '"checkpoint_tso":' 449348961771454469, '"checkpoint_time":' '"2024-04-26' '19:17:35.781",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] start tidb cluster in /tmp/tidb_cdc_test/multi_source Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table processor_delay.t19 not exists for 1-th check, retry later [2024/04/26 19:17:36.820 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] [2024/04/26 19:17:37.160 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.12162.out cli unsafe delete-service-gc-safepoint table test.finish_mark not exists for 21-th check, retry later Confirm that you know what this command will do and use it at your own risk [Y/N] CDC service GC safepoint truncated in PD! PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:17:37.948 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] Verifying downstream PD is started... table mark.finish_mark_3 not exists for 25-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:38 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041cab40 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348962065580035 MinTableBarrierTs:449348962065580036 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348962327461891, checkpointTs: 449348962327461891, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348962327461892, checkpointTs: 449348962327461892, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348962327461891, checkpointTs: 449348962327461891, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e1d65f5-4fd0-484e-8968-b3452e1a672b {"id":"7e1d65f5-4fd0-484e-8968-b3452e1a672b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2c88 7e1d65f5-4fd0-484e-8968-b3452e1a672b /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348962065580036,"min-table-barrier-ts":449348962327461892,"admin-job-type":0} /tidb/cdc/default/default/task/position/7e1d65f5-4fd0-484e-8968-b3452e1a672b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041cab40 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348962065580035 MinTableBarrierTs:449348962065580036 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348962327461891, checkpointTs: 449348962327461891, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348962327461892, checkpointTs: 449348962327461892, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348962327461891, checkpointTs: 449348962327461891, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e1d65f5-4fd0-484e-8968-b3452e1a672b {"id":"7e1d65f5-4fd0-484e-8968-b3452e1a672b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2c88 7e1d65f5-4fd0-484e-8968-b3452e1a672b /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348962065580036,"min-table-barrier-ts":449348962327461892,"admin-job-type":0} /tidb/cdc/default/default/task/position/7e1d65f5-4fd0-484e-8968-b3452e1a672b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-p+ grep -q 'failed to get info:' ath":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0041cab40 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348962065580035 MinTableBarrierTs:449348962065580036 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348962327461891, checkpointTs: 449348962327461891, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348962327461892, checkpointTs: 449348962327461892, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348962327461891, checkpointTs: 449348962327461891, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e1d65f5-4fd0-484e-8968-b3452e1a672b {"id":"7e1d65f5-4fd0-484e-8968-b3452e1a672b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2c88 7e1d65f5-4fd0-484e-8968-b3452e1a672b /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348962065580036,"min-table-barrier-ts":449348962327461892,"admin-job-type":0} /tidb/cdc/default/default/task/position/7e1d65f5-4fd0-484e-8968-b3452e1a672b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' table multi_rocks.finish_mark not exists for 4-th check, retry later [2024/04/26 19:17:38.832 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark15000] [2024/04/26 19:17:38.848 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark15000] [ts=449348962115911682] [2024/04/26 19:17:38.848 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark15000] [2024/04/26 19:17:38.848 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348962115911682] [2024/04/26 19:17:38.850 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:38.853 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:38.857 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:38.862 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:38.862 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark18000] table processor_delay.t19 exists table processor_delay.t20 not exists for 1-th check, retry later + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.12197.out cli unsafe reset --no-confirm --pd=http://127.0.0.1:2379 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_only_block_related_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/move_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... reset and all metadata truncated in PD! PASS coverage: 2.1% of statements in github.com/pingcap/tiflow/... [2024/04/26 19:17:39.171 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] "is-owner": true, run task successfully table test.finish_mark not exists for 22-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/move_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table mark.finish_mark_3 not exists for 26-th check, retry later [2024/04/26 19:17:40.032 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table multi_rocks.finish_mark not exists for 5-th check, retry later Run finished, takes 37.204725321s INSERT - Takes(s): 37.2, Count: 50000, OPS: 1344.1, Avg(us): 7388, Min(us): 735, Max(us): 363934, 95th(us): 81000, 99th(us): 88000 table consistent_replicate_nfs.usertable exists wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... + set +x [2024/04/26 19:17:40.862 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] table processor_delay.t20 exists table processor_delay.t21 not exists for 1-th check, retry later wait process cdc.test exit for 3-th time... Verifying downstream PD is started... owner pid: 4562 owner id 7e1d65f5-4fd0-484e-8968-b3452e1a672b default owner key /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2c88 [Fri Apr 26 19:17:40 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.46854687.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_gap_between_watch_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_gap_between_watch_capture.server2 --cluster-id default --addr 127.0.0.1:8301 + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:17:41.184 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark16000] [2024/04/26 19:17:41.320 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark16000] [ts=449348963033415685] [2024/04/26 19:17:41.321 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark16000] [2024/04/26 19:17:41.321 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348963033415685] [2024/04/26 19:17:41.323 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:41.328 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:41.331 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:41.335 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:41.336 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit table test.finish_mark not exists for 23-th check, retry later table mark.finish_mark_3 not exists for 27-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688037800014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:5777, start at 2024-04-26 19:17:39.471987265 +0800 CST m=+7.833669724 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:39.489 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:39.473 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:39.473 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688037800014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:5777, start at 2024-04-26 19:17:39.471987265 +0800 CST m=+7.833669724 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:39.489 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:39.473 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:39.473 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688037e40003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-ws663, pid:5806, start at 2024-04-26 19:17:39.45156937 +0800 CST m=+7.694832655 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:39.467 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:39.449 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:39.449 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/common_1/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/common_1/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/common_1/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/common_1/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/common_1/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table multi_rocks.finish_mark not exists for 6-th check, retry later [2024/04/26 19:17:42.044 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] start tidb cluster in /tmp/tidb_cdc_test/ddl_only_block_related_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [2024/04/26 19:17:42.930 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] [Pipeline] // dir table processor_delay.t21 exists table processor_delay.t22 not exists for 1-th check, retry later [Pipeline] } [Pipeline] // withCredentials [Pipeline] } Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.12289.out cli unsafe resolve-lock --region=46 table test.finish_mark not exists for 24-th check, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } PASS coverage: 2.0% of statements in github.com/pingcap/tiflow/... [2024/04/26 19:17:44.057 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark17000] [2024/04/26 19:17:44.073 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark17000] [ts=449348963033415685] [2024/04/26 19:17:44.073 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark17000] [2024/04/26 19:17:44.073 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348963033415685] [2024/04/26 19:17:44.075 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:44.078 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:44.081 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:44.085 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:44.085 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark19000] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:43 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00311ba70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348963638181893 MinTableBarrierTs:449348963638181895 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e1d65f5-4fd0-484e-8968-b3452e1a672b {"id":"7e1d65f5-4fd0-484e-8968-b3452e1a672b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/e5b957cb-a8d6-4b9d-a415-98d650df32af {"id":"e5b957cb-a8d6-4b9d-a415-98d650df32af","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2c88 7e1d65f5-4fd0-484e-8968-b3452e1a672b /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2cc1 e5b957cb-a8d6-4b9d-a415-98d650df32af /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348963638181893,"min-table-barrier-ts":449348963900325895,"admin-job-type":0} /tidb/cdc/default/default/task/position/7e1d65f5-4fd0-484e-8968-b3452e1a672b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b957cb-a8d6-4b9d-a415-98d650df32af/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00311ba70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348963638181893 MinTableBarrierTs:449348963638181895 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e1d65f5-4fd0-484e-8968-b3452e1a672b {"id":"7e1d65f5-4fd0-484e-8968-b3452e1a672b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/e5b957cb-a8d6-4b9d-a415-98d650df32af {"id":"e5b957cb-a8d6-4b9d-a415-98d650df32af","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2c88 7e1d65f5-4fd0-484e-8968-b3452e1a672b /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2cc1 e5b957cb-a8d6-4b9d-a415-98d650df32af /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348963638181893,"min-table-barrier-ts":449348963900325895,"admin-job-type":0} /tidb/cdc/default/default/task/position/7e1d65f5-4fd0-484e-8968-b3452e1a672b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b957cb-a8d6-4b9d-a415-98d650df32af/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00311ba70 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348963638181893 MinTableBarrierTs:449348963638181895 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e1d65f5-4fd0-484e-8968-b3452e1a672b {"id":"7e1d65f5-4fd0-484e-8968-b3452e1a672b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/e5b957cb-a8d6-4b9d-a415-98d650df32af {"id":"e5b957cb-a8d6-4b9d-a415-98d650df32af","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2c88 7e1d65f5-4fd0-484e-8968-b3452e1a672b /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2cc1 e5b957cb-a8d6-4b9d-a415-98d650df32af /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348963638181893,"min-table-barrier-ts":449348963900325895,"admin-job-type":0} /tidb/cdc/default/default/task/position/7e1d65f5-4fd0-484e-8968-b3452e1a672b/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e5b957cb-a8d6-4b9d-a415-98d650df32af/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep -v "7e1d65f5-4fd0-484e-8968-b3452e1a672b default" | grep id table mark.finish_mark_3 not exists for 28-th check, retry later table multi_rocks.finish_mark exists check diff successfully + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.cli.7023.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 1-th time... [2024/04/26 19:17:44.944 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] "id": "e5b957cb-a8d6-4b9d-a415-98d650df32af", run task successfully + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cli.cli.12316.out cli unsafe resolve-lock --region=46 --ts=449348963069067270 table processor_delay.t22 exists table processor_delay.t23 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release PASS table test.finish_mark not exists for 25-th check, retry later coverage: 2.0% of statements in github.com/pingcap/tiflow/... table mark.finish_mark_3 not exists for 29-th check, retry later [2024/04/26 19:17:46.086 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark19000] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:17:46 CST 2024] <<<<<< run test case multi_rocks success! >>>>>> capture_id: e5b957cb-a8d6-4b9d-a415-98d650df32af + set +x + tso='449348964128653313 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + awk -F ' ' '{print $1}' + echo 449348964128653313 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + set +x [Fri Apr 26 19:17:46 CST 2024] <<<<<< START cdc server in common_1 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.70417043.out server --log-file /tmp/tidb_cdc_test/common_1/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/common_1/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 1 ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep 'e5b957cb-a8d6-4b9d-a415-98d650df32af' e5b957cb-a8d6-4b9d-a415-98d650df32af run task successfully curl -X GET http://127.0.0.1:8300/status | grep '"is_owner": false' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 176 100 176 0 0 20384 0 --:--:-- --:--:-- --:--:-- 22000 "is_owner": false run task successfully Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore + set +x % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 218 100 212 100 6 106k 3100 --:--:-- --:--:-- --:--:-- 207k { "error_msg": "[CDC:ErrAPIInvalidParam]invalid log level: json: cannot unmarshal string into Go value of type struct { Level string \"json:\\\"log_level\\\"\" }", "error_code": "CDC:ErrAPIInvalidParam" [2024/04/26 19:17:47.118 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t23 exists table processor_delay.t24 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 26-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [2024/04/26 19:17:48.099 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark19000] [Pipeline] // stage ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] } [Pipeline] // container [Pipeline] } table mark.finish_mark_3 not exists for 30-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_resume_with_checkpoint_ts/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [2024/04/26 19:17:49.130 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark18000] [2024/04/26 19:17:49.147 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark18000] [ts=449348964881530885] [2024/04/26 19:17:49.147 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark18000] [2024/04/26 19:17:49.147 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348964881530885] [2024/04/26 19:17:49.149 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:49.152 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:49.157 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:49.161 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:49.161 +08:00] [INFO] [case.go:571] ["downstream start wait for table"] [tblName=finishmark20000] [Pipeline] // withEnv [Pipeline] } table processor_delay.t24 exists table processor_delay.t25 not exists for 1-th check, retry later [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:17:49 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d052f7aa-e5e8-46b6-af1b-f18199bb52a0 {"id":"d052f7aa-e5e8-46b6-af1b-f18199bb52a0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdf28f7 d052f7aa-e5e8-46b6-af1b-f18199bb52a0 /tidb/cdc/default/default/upstream/7362133357703055558 {"id":7362133357703055558,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d052f7aa-e5e8-46b6-af1b-f18199bb52a0 {"id":"d052f7aa-e5e8-46b6-af1b-f18199bb52a0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdf28f7 d052f7aa-e5e8-46b6-af1b-f18199bb52a0 /tidb/cdc/default/default/upstream/7362133357703055558 {"id":7362133357703055558,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d052f7aa-e5e8-46b6-af1b-f18199bb52a0 {"id":"d052f7aa-e5e8-46b6-af1b-f18199bb52a0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1fdf28f7 d052f7aa-e5e8-46b6-af1b-f18199bb52a0 /tidb/cdc/default/default/upstream/7362133357703055558 {"id":7362133357703055558,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.common_1.cli.7080.out cli changefeed create --start-ts=449348964128653313 --sink-uri=mysql://root@127.0.0.1:3306/ ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 27-th check, retry later [2024/04/26 19:17:50.135 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark19000] nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table mark.finish_mark_3 not exists for 31-th check, retry later } % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 176 100 176 0 0 191k 0 --:--:-- --:--:-- --:--:-- 171k { "version": "v7.5.1-57-g10f68e69e", "git_hash": "10f68e69e5863d5c0b43c90372b2921ea2b0650f", "id": "c990d542-4c5a-4e69-a201-0788840b9152", "pid": 11501, "is_owner": true Create changefeed successfully! ID: 19211d62-1df8-46ab-8915-525134517af9 Info: {"upstream_id":7362133357703055558,"namespace":"default","id":"19211d62-1df8-46ab-8915-525134517af9","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-04-26T19:17:50.595778144+08:00","start_ts":449348964128653313,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348964128653313,"checkpoint_ts":449348964128653313,"checkpoint_time":"2024-04-26 19:17:44.773"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... }wait process cdc.test exit for 1-th time... [2024/04/26 19:17:51.161 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark20000] table processor_delay.t25 exists table processor_delay.t26 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/consistent_replicate_nfs/sync_diff/output" source-instances = ["mysql1"] target-instance = "tidb0" target-check-tables = ["consistent_replicate_nfs.usertable*"] [data-sources] [data-sources.mysql1] host = "127.0.0.1" port = 4000 user = "root" password = "" snapshot = "449348962076327992" [data-sources.tidb0] host = "127.0.0.1" port = 3306 user = "root" password = "" cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:17:51 CST 2024] <<<<<< run test case cli success! >>>>>> [2024/04/26 19:17:51.730 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/04/26 19:17:51.730 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/04/26 19:17:51.731 +08:00] [INFO] [redo.go:163] ["apply redo log starts"] [checkpointTs=449348961683112327] [resolvedTs=449348962076327992] [2024/04/26 19:17:51.731 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=262.393ยตs] [2024/04/26 19:17:51.731 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:17:51.731 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:17:51.736 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22NO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:17:51.739 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/04/26 19:17:51.739 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/04/26 19:17:51.740 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/04/26 19:17:51.740 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/04/26 19:17:51.740 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/04/26 19:17:51.740 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/04/26 19:17:51.740 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/04/26 19:17:51.741 +08:00] [INFO] [file.go:284] ["ignore logs which commitTs is greater than resolvedTs"] [filename=127.0.0.1:8300_debe933a-b456-4b01-be72-2c381a0cfee5_row_449348962521972839_2a52f652-1cb7-4c53-be4a-d33cd74c754d.log] [endTs=449348962076327992] [2024/04/26 19:17:51.741 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:17:51.741 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:17:51.745 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CNO_AUTO_VALUE_ON_ZERO%2CALLOW_INVALID_DATES%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:17:51.746 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/04/26 19:17:51.755 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=24.908307ms] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 28-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later [2024/04/26 19:17:52.062 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}] [checkpointTs=449348962076327992] [2024/04/26 19:17:52.062 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}] [checkpointTs=449348962076327992] [2024/04/26 19:17:52.162 +08:00] [INFO] [redo.go:219] ["apply redo log finishes"] [appliedLogCount=2001] [appliedDDLCount=0] [currentCheckpoint=449348962076327992] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/04/26 19:17:52.162 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] Apply redo log successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff successfully [Fri Apr 26 19:17:52 CST 2024] <<<<<< run test case consistent_replicate_nfs success! >>>>>> count(*) 47557 table mark.finish_mark_3 not exists for 32-th check, retry later [2024/04/26 19:17:52.149 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark19000] [2024/04/26 19:17:52.221 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark19000] [ts=449348965799034885] [2024/04/26 19:17:52.221 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark19000] [2024/04/26 19:17:52.221 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348965799034885] [2024/04/26 19:17:52.224 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:52.227 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:52.231 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:52.235 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:52.236 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] + set +x table processor_delay.t26 exists table processor_delay.t27 not exists for 1-th check, retry later [2024/04/26 19:17:53.173 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark20000] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68811648000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:16103, start at 2024-04-26 19:17:53.709477117 +0800 CST m=+8.370996078 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:53.724 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:53.732 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:53.732 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68811648000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:16103, start at 2024-04-26 19:17:53.709477117 +0800 CST m=+8.370996078 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:53.724 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:53.732 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:53.732 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68810ac00002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:16140, start at 2024-04-26 19:17:52.948876102 +0800 CST m=+7.454115116 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:52.973 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:52.944 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:52.944 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table test.finish_mark not exists for 29-th check, retry later Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 33-th check, retry later Logging trace to /tmp/tidb_cdc_test/move_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/move_table/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/move_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/move_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t27 exists table processor_delay.t28 not exists for 1-th check, retry later [2024/04/26 19:17:55.187 +08:00] [INFO] [case.go:638] ["try get end ts"] [query="SELECT JOB_ID FROM information_schema.ddl_jobs WHERE table_name = ?"] [tableName=finishmark20000] [2024/04/26 19:17:55.204 +08:00] [INFO] [case.go:673] ["get end ts"] [tableName=finishmark20000] [ts=449348966716538885] [2024/04/26 19:17:55.204 +08:00] [INFO] [case.go:576] ["ddl synced"] [table=finishmark20000] [2024/04/26 19:17:55.204 +08:00] [INFO] [case.go:582] ["downstream sync success"] [endTs=449348966716538885] [2024/04/26 19:17:55.206 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:55.209 +08:00] [INFO] [case.go:186] ["sequence verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:55.213 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:4000)/bank] [2024/04/26 19:17:55.217 +08:00] [INFO] [case.go:303] ["bank verify pass"] [tag=root@tcp(127.0.0.1:3306)/bank] [2024/04/26 19:17:55.217 +08:00] [INFO] [case.go:603] ["one downstream exit due to receive done"] [2024/04/26 19:17:55.217 +08:00] [INFO] [case.go:615] ["bank test finished"] [valid=20] [tries=20] [ratio=1] + cleanup_process cdc.test table test.finish_mark not exists for 30-th check, retry later wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68811764000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:6172, start at 2024-04-26 19:17:53.795216896 +0800 CST m=+10.770447307 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:53.818 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:53.803 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:53.803 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68811764000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:6172, start at 2024-04-26 19:17:53.795216896 +0800 CST m=+10.770447307 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:53.818 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:53.803 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:53.803 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68811b140005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-4qcbz, pid:6206, start at 2024-04-26 19:17:54.001473265 +0800 CST m=+10.834795015 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:54.026 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:54.039 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:54.039 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) table mark.finish_mark_3 not exists for 34-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully nonempty select id, val from test.availability1 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later [Pipeline] // cache [Pipeline] } + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.17338.out cli tso query --pd=http://127.0.0.1:2379 wait process cdc.test exit for 2-th time... Logging trace to /tmp/tidb_cdc_test/multi_source/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_source/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_source/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } start tidb cluster in /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts Starting Upstream PD... [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit ++ date + echo '[Fri Apr 26 19:17:56 CST 2024] <<<<<< run test case bank success! >>>>>>' [Fri Apr 26 19:17:56 CST 2024] <<<<<< run test case bank success! >>>>>> + stop_tidb_cluster Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table test.finish_mark not exists for 31-th check, retry later table processor_delay.t28 exists table processor_delay.t29 not exists for 1-th check, retry later skip some SQLs in tidb v4.0.x nonempty select id, val from test.availability1 where id=1 and val=22 run task successfully empty select id, val from test.availability1 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.7407.out cli tso query --pd=http://127.0.0.1:2379 table common_1.v1 exists table common_1.recover_and_insert not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68813aa40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:5829, start at 2024-04-26 19:17:56.050811937 +0800 CST m=+8.728437894 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:56.065 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:56.058 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:56.058 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68813aa40014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:5829, start at 2024-04-26 19:17:56.050811937 +0800 CST m=+8.728437894 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:56.065 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:56.058 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:56.058 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68813bdc0001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:5856, start at 2024-04-26 19:17:56.088001686 +0800 CST m=+8.624274753 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:19:56.109 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:17:56.087 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:07:56.087 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_only_block_related_table/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_3 not exists for 35-th check, retry later + set +x + tso='449348967303217153 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348967303217153 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x ***************** properties ***************** "insertproportion"="0" "recordcount"="10000" "mysql.db"="move_table" "requestdistribution"="uniform" "threadcount"="10" "dotransactions"="false" "scanproportion"="0" "operationcount"="0" "readallfields"="true" "mysql.port"="4000" "readproportion"="0" "updateproportion"="0" "workload"="core" "mysql.host"="127.0.0.1" "mysql.user"="root" ********************************************** table processor_delay.t29 exists table processor_delay.t30 not exists for 1-th check, retry later table test.finish_mark not exists for 32-th check, retry later table common_1.recover_and_insert not exists for 2-th check, retry later table mark.finish_mark_3 not exists for 36-th check, retry later empty select id, val from test.availability1 where id=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later + set +x + tso='449348967819902977 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348967819902977 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:18:00 CST 2024] <<<<<< START cdc server in multi_source case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.74317433.out server --log-file /tmp/tidb_cdc_test/multi_source/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_source/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [Fri Apr 26 19:18:00 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.70557057.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table test.finish_mark not exists for 1-th check, retry later table test.finish_mark not exists for 2-th check, retry later table test.finish_mark not exists for 3-th check, retry later table test.finish_mark not exists for 4-th check, retry later table test.finish_mark not exists for 5-th check, retry later table processor_delay.t30 exists table processor_delay.t31 not exists for 1-th check, retry later table test.finish_mark not exists for 33-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_storage_file/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table mark.finish_mark_3 not exists for 37-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) nonempty select id, val from test.availability2 where id=1 and val=1 run task successfully nonempty select id, val from test.availability2 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table test.finish_mark not exists for 6-th check, retry later table common_1.recover_and_insert not exists for 3-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > table processor_delay.t31 exists table processor_delay.t32 exists table processor_delay.t33 not exists for 1-th check, retry later < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:03 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/439000bc-f2aa-4a0e-bff6-a051ea7d8e4e {"id":"439000bc-f2aa-4a0e-bff6-a051ea7d8e4e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a201851f5 439000bc-f2aa-4a0e-bff6-a051ea7d8e4e /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/439000bc-f2aa-4a0e-bff6-a051ea7d8e4e {"id":"439000bc-f2aa-4a0e-bff6-a051ea7d8e4e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a201851f5 439000bc-f2aa-4a0e-bff6-a051ea7d8e4e /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/439000bc-f2aa-4a0e-bff6-a051ea7d8e4e {"id":"439000bc-f2aa-4a0e-bff6-a051ea7d8e4e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a201851f5 439000bc-f2aa-4a0e-bff6-a051ea7d8e4e /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.cli.7095.out cli changefeed create --sink-uri=mysql://root@127.0.0.1:3306/ -c=ddl-only-block-related-table table test.finish_mark not exists for 34-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:03 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03e7555b-a8f0-4386-b5a4-ec351e6e7868 {"id":"03e7555b-a8f0-4386-b5a4-ec351e6e7868","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a200399fd 03e7555b-a8f0-4386-b5a4-ec351e6e7868 /tidb/cdc/default/default/upstream/7362133397393262633 {"id":7362133397393262633,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03e7555b-a8f0-4386-b5a4-ec351e6e7868 {"id":"03e7555b-a8f0-4386-b5a4-ec351e6e7868","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a200399fd 03e7555b-a8f0-4386-b5a4-ec351e6e7868 /tidb/cdc/default/default/upstream/7362133397393262633 {"id":7362133397393262633,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/03e7555b-a8f0-4386-b5a4-ec351e6e7868 {"id":"03e7555b-a8f0-4386-b5a4-ec351e6e7868","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a200399fd 03e7555b-a8f0-4386-b5a4-ec351e6e7868 /tidb/cdc/default/default/upstream/7362133397393262633 {"id":7362133397393262633,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_source.cli.7468.out cli changefeed create --start-ts=449348967819902977 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: ddl-only-block-related-table Info: {"upstream_id":7362133426535730208,"namespace":"default","id":"ddl-only-block-related-table","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-04-26T19:18:04.112622636+08:00","start_ts":449348969158148100,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348969158148100,"checkpoint_ts":449348969158148100,"checkpoint_time":"2024-04-26 19:18:03.959"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... Run finished, takes 4.732967093s INSERT - Takes(s): 4.7, Count: 9998, OPS: 2129.2, Avg(us): 4619, Min(us): 1604, Max(us): 38798, 95th(us): 9000, 99th(us): 13000 [Fri Apr 26 19:18:03 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.1741017412.out server --log-file /tmp/tidb_cdc_test/move_table/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data1 --cluster-id default --addr 127.0.0.1:8300 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_3 not exists for 38-th check, retry later nonempty select id, val from test.availability2 where id=1 and val=22 run task successfully empty select id, val from test.availability2 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later Create changefeed successfully! ID: fdddda3f-63e2-47b3-a400-477d52edb75f Info: {"upstream_id":7362133397393262633,"namespace":"default","id":"fdddda3f-63e2-47b3-a400-477d52edb75f","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:18:04.308337944+08:00","start_ts":449348967819902977,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348967819902977,"checkpoint_ts":449348967819902977,"checkpoint_time":"2024-04-26 19:17:58.854"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table common_1.recover_and_insert not exists for 4-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_file Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 7-th check, retry later table processor_delay.t33 exists table processor_delay.t34 not exists for 1-th check, retry later table test.finish_mark not exists for 35-th check, retry later + set +x table mark.finish_mark_3 not exists for 39-th check, retry later [2024/04/26 19:17:59.773 +08:00] [WARN] [diff.go:551] ["checksum is not equal"] [table=`test`.`many_cols`] [where="((TRUE) AND TRUE)"] ["source checksum"=2660252693] ["target checksum"=1858657880] ["get source checksum cost"=1.713447ms] ["get target checksum cost"=1.250997ms] [2024/04/26 19:17:59.775 +08:00] [WARN] [diff.go:893] ["find different row"] [column=val] [row1="{ id: 1, val: 3610, }"] [row2="{ id: 1, val: 3363, }"] [2024/04/26 19:17:59.860 +08:00] [WARN] [diff.go:745] ["rows is not equal"] [table=`test`.`many_cols`] [where="((TRUE) AND TRUE)"] [cost=86.934299ms] [2024/04/26 19:17:59.862 +08:00] [WARN] [diff.go:384] ["check chunk data not equal"] [chunk="{\"id\":0,\"bounds\":[],\"where\":\"((TRUE) AND TRUE)\",\"args\":null,\"state\":\"failed\"}"] + set +x go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/pingcap/errors v0.11.5-0.20240318064555-6bd07397691f go: downloading github.com/pingcap/tidb-tools v0.0.0-20240408054520-d28ee6cd546e go: downloading github.com/BurntSushi/toml v1.3.2 go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20240425142112-54ba0ed68407 go: downloading go.uber.org/atomic v1.11.0 go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/pingcap/failpoint v0.0.0-20240412033321-fd0796e60f86 go: downloading github.com/pingcap/tidb/pkg/parser v0.0.0-20240425142112-54ba0ed68407 go: downloading github.com/coreos/go-semver v0.3.1 go: downloading github.com/go-sql-driver/mysql v1.7.1 go: downloading google.golang.org/grpc v1.62.1 empty select id, val from test.availability2 where id=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=1 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/savepoint/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table common_1.recover_and_insert exists table common_1.finish_mark not exists for 1-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:06 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.cli.17447.out cli changefeed create --start-ts=449348967303217153 '--sink-uri=mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1' =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kill_owner_with_ddl/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/protobuf v1.33.0 go: downloading golang.org/x/sys v0.19.0 go: downloading golang.org/x/net v0.24.0 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 table test.finish_mark not exists for 8-th check, retry later Create changefeed successfully! ID: 21eb0bcd-3128-4bef-8bae-9c18b3b46ddc Info: {"upstream_id":7362133410276671937,"namespace":"default","id":"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:18:07.415727236+08:00","start_ts":449348967303217153,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348967303217153,"checkpoint_ts":449348967303217153,"checkpoint_time":"2024-04-26 19:17:56.883"} PASS ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading golang.org/x/text v0.14.0 coverage: 2.4% of statements in github.com/pingcap/tiflow/... table test.finish_mark not exists for 36-th check, retry later table processor_delay.t34 not exists for 2-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table mark.finish_mark_3 not exists for 40-th check, retry later table common_1.finish_mark not exists for 2-th check, retry later nonempty select id, val from test.availability3 where id=1 and val=1 run task successfully nonempty select id, val from test.availability3 where id=1 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table ddl_only_block_related_table.finish_mark not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/savepoint Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... + set +x [Fri Apr 26 19:18:09 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.1747417476.out server --log-file /tmp/tidb_cdc_test/move_table/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data2 --cluster-id default --addr 127.0.0.1:8301 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/kill_owner_with_ddl Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6881f044001d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:6400, start at 2024-04-26 19:18:07.670532654 +0800 CST m=+5.195469693 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:07.677 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:07.684 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:07.684 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6881f044001d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:6400, start at 2024-04-26 19:18:07.670532654 +0800 CST m=+5.195469693 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:07.677 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:07.684 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:07.684 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6881f1c00015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-50w1g, pid:6475, start at 2024-04-26 19:18:07.763976019 +0800 CST m=+5.219628240 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:07.773 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:07.778 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:07.778 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 9-th check, retry later table test.finish_mark not exists for 37-th check, retry later table processor_delay.t34 exists table processor_delay.t35 exists table processor_delay.t36 not exists for 1-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table common_1.finish_mark exists table ddl_only_block_related_table.finish_mark not exists for 2-th check, retry later Verifying downstream PD is started... check diff successfully table mark.finish_mark_3 not exists for 41-th check, retry later go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 go: downloading golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f go: downloading go.etcd.io/etcd/client/v3 v3.5.12 go: downloading github.com/tikv/client-go/v2 v2.0.8-0.20240424052342-0229f4077f0c go: downloading golang.org/x/sync v0.7.0 go: downloading github.com/docker/go-units v0.5.0 go: downloading github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754 go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7 go: downloading github.com/opentracing/opentracing-go v1.2.0 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/prometheus/client_golang v1.19.0 go: downloading github.com/tikv/pd/client v0.0.0-20240322051414-fb9e2d561b6e go: downloading github.com/tidwall/btree v1.7.0 go: downloading cloud.google.com/go/storage v1.36.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.1 go: downloading github.com/pingcap/tipb v0.0.0-20240318032315-55a7867ddd50 go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 go: downloading github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581 go: downloading github.com/aws/aws-sdk-go v1.50.0 go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1 go: downloading github.com/go-resty/resty/v2 v2.11.0 go: downloading github.com/klauspost/compress v1.17.7 go: downloading github.com/ks3sdklib/aws-sdk-go v1.2.9 go: downloading github.com/tikv/pd v1.1.0-beta.0.20230203015356-248b3f0be132 go: downloading github.com/spf13/pflag v1.0.5 go: downloading golang.org/x/oauth2 v0.18.0 go: downloading google.golang.org/api v0.162.0 go: downloading cloud.google.com/go v0.112.0 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading go.etcd.io/etcd/api/v3 v3.5.12 go: downloading go.uber.org/mock v0.4.0 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading github.com/jfcg/sorty/v2 v2.1.0 go: downloading github.com/coocood/freecache v1.2.1 go: downloading github.com/cockroachdb/pebble v1.1.0 go: downloading golang.org/x/tools v0.20.0 go: downloading github.com/uber/jaeger-client-go v2.30.0+incompatible go: downloading github.com/carlmjohnson/flagext v0.21.0 go: downloading github.com/google/btree v1.1.2 nonempty select id, val from test.availability3 where id=1 and val=22 run task successfully empty select id, val from test.availability3 where id=1 TEST FAILED: OUTPUT CONTAINS 'id:' ____________________________________ *************************** 1. row *************************** id: 1 val: 22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later go: downloading golang.org/x/time v0.5.0 go: downloading github.com/dolthub/swiss v0.2.1 go: downloading github.com/dgraph-io/ristretto v0.1.1 go: downloading github.com/twmb/murmur3 v1.1.6 go: downloading github.com/danjacques/gofslock v0.0.0-20220131014315-6e321f4509c8 go: downloading github.com/influxdata/tdigest v0.0.1 go: downloading github.com/golang/snappy v0.0.4 go: downloading github.com/opentracing/basictracer-go v1.1.0 go: downloading github.com/stretchr/testify v1.9.0 go: downloading github.com/pingcap/sysutil v1.0.1-0.20240311050922-ae81ee01f3a5 go: downloading github.com/shirou/gopsutil/v3 v3.24.2 go: downloading github.com/jellydator/ttlcache/v3 v3.0.1 go: downloading github.com/gorilla/mux v1.8.0 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v1.5.1 go: downloading github.com/AzureAD/microsoft-authentication-library-for-go v1.2.1 go: downloading golang.org/x/crypto v0.22.0 go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.12 go: downloading github.com/prometheus/client_model v0.6.1 go: downloading github.com/jedib0t/go-pretty/v6 v6.2.2 go: downloading github.com/cloudfoundry/gosigar v1.3.6 go: downloading github.com/joho/sqltocsv v0.0.0-20210428211105-a6d6801d59df go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.53.0 go: downloading github.com/prometheus/procfs v0.13.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/pkg/errors v0.9.1 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading github.com/spkg/bom v1.0.0 go: downloading github.com/xitongsys/parquet-go v1.6.0 wait process cdc.test exit for 1-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release go: downloading cloud.google.com/go/compute v1.24.0 go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 go: downloading github.com/google/pprof v0.0.0-20240117000934-35fc243c5815 go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f go: downloading github.com/yangkeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117 go: downloading github.com/dolthub/maphash v0.1.0 go: downloading github.com/cockroachdb/errors v1.11.1 go: downloading github.com/jfcg/sixb v1.3.8 go: downloading github.com/otiai10/copy v1.2.0 go: downloading github.com/lestrrat-go/jwx/v2 v2.0.21 go: downloading cloud.google.com/go/iam v1.1.6 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 table test.finish_mark not exists for 10-th check, retry later go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 go: downloading github.com/cheggaaa/pb/v3 v3.0.8 go: downloading github.com/coreos/go-systemd/v22 v22.5.0 go: downloading github.com/kylelemons/godebug v1.1.0 go: downloading github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c go: downloading github.com/Azure/go-ntlmssp v0.0.0-20221128193559-754e69321358 go: downloading github.com/go-asn1-ber/asn1-ber v1.5.4 go: downloading github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible go: downloading github.com/tklauser/go-sysconf v0.3.12 go: downloading github.com/robfig/cron/v3 v3.0.1 go: downloading github.com/mattn/go-runewidth v0.0.15 go: downloading go.opencensus.io v0.23.1-0.20220331163232-052120675fac go: downloading go.opentelemetry.io/otel v1.22.0 go: downloading github.com/robfig/cron v1.2.0 go: downloading go.opentelemetry.io/otel/trace v1.22.0 go: downloading github.com/apache/thrift v0.16.0 table processor_delay.t36 not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:18:11 CST 2024] <<<<<< START cdc server in changefeed_resume_with_checkpoint_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_resume_with_checkpoint_ts.77147716.out server --log-file /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/VividCortex/ewma v1.2.0 go: downloading github.com/fatih/color v1.16.0 go: downloading github.com/mattn/go-colorable v0.1.13 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/golang/glog v1.2.0 go: downloading github.com/golang-jwt/jwt/v5 v5.2.0 go: downloading github.com/pingcap/badger v1.5.1-0.20230103063557-828f39b09b6d go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 go: downloading github.com/rivo/uniseg v0.4.6 go: downloading github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b go: downloading github.com/cockroachdb/redact v1.1.5 go: downloading github.com/getsentry/sentry-go v0.27.0 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/lestrrat-go/blackmagic v1.0.2 go: downloading github.com/lestrrat-go/httprc v1.0.5 go: downloading github.com/lestrrat-go/iter v1.0.2 go: downloading github.com/tklauser/numcpus v0.6.1 go: downloading github.com/lestrrat-go/option v1.0.1 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading github.com/lestrrat-go/httpcc v1.0.1 go: downloading github.com/rogpeppe/go-internal v1.12.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2 go: downloading github.com/ncw/directio v1.0.5 go: downloading github.com/klauspost/cpuid v1.3.1 go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64 table test.finish_mark not exists for 38-th check, retry later wait process cdc.test exit for 3-th time... go: downloading github.com/go-logr/logr v1.4.1 go: downloading go.opentelemetry.io/otel/metric v1.22.0 go: downloading github.com/go-logr/stdr v1.2.2 table ddl_only_block_related_table.finish_mark not exists for 3-th check, retry later table mark.finish_mark_3 not exists for 42-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:18:12 CST 2024] <<<<<< run test case common_1 success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:12 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {UpstreamID:7362133410276671937 Namespace:default ID:21eb0bcd-3128-4bef-8bae-9c18b3b46ddc SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:07.415727236 +0800 CST StartTs:449348967303217153 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003274900 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348970055467012} {CheckpointTs:449348967905886301 MinTableBarrierTs:449348971156471809 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9af6bce4-b810-45a5-a132-1b560a622f4a {"id":"9af6bce4-b810-45a5-a132-1b560a622f4a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb5b 9af6bce4-b810-45a5-a132-1b560a622f4a /tidb/cdc/default/default/changefeed/info/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"upstream-id":7362133410276671937,"namespace":"default","changefeed-id":"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:07.415727236+08:00","start-ts":449348967303217153,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348970055467012} /tidb/cdc/default/default/changefeed/status/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":449348967905886301,"min-table-barrier-ts":449348971156471809,"admin-job-type":0} /tidb/cdc/default/default/task/position/962197a4-05d7-418c-856b-5c08556d01ee/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9af6bce4-b810-45a5-a132-1b560a622f4a/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {UpstreamID:7362133410276671937 Namespace:default ID:21eb0bcd-3128-4bef-8bae-9c18b3b46ddc SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:07.415727236 +0800 CST StartTs:449348967303217153 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003274900 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348970055467012} {CheckpointTs:449348967905886301 MinTableBarrierTs:449348971156471809 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9af6bce4-b810-45a5-a132-1b560a622f4a {"id":"9af6bce4-b810-45a5-a132-1b560a622f4a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb5b 9af6bce4-b810-45a5-a132-1b560a622f4a /tidb/cdc/default/default/changefeed/info/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"upstream-id":7362133410276671937,"namespace":"default","changefeed-id":"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:07.415727236+08:00","start-ts":449348967303217153,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348970055467012} /tidb/cdc/default/default/changefeed/status/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":449348967905886301,"min-table-barrier-ts":449348971156471809,"admin-job-type":0} /tidb/cdc/default/default/task/position/962197a4-05d7-418c-856b-5c08556d01ee/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9af6bce4-b810-45a5-a132-1b560a622f4a/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {UpstreamID:7362133410276671937 Namespace:default ID:21eb0bcd-3128-4bef-8bae-9c18b3b46ddc SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:07.415727236 +0800 CST StartTs:449348967303217153 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003274900 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348970055467012} {CheckpointTs:449348967905886301 MinTableBarrierTs:449348971156471809 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9af6bce4-b810-45a5-a132-1b560a622f4a {"id":"9af6bce4-b810-45a5-a132-1b560a622f4a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb5b 9af6bce4-b810-45a5-a132-1b560a622f4a /tidb/cdc/default/default/changefeed/info/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"upstream-id":7362133410276671937,"namespace":"default","changefeed-id":"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:07.415727236+08:00","start-ts":449348967303217153,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348970055467012} /tidb/cdc/default/default/changefeed/status/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":449348967905886301,"min-table-barrier-ts":449348971156471809,"admin-job-type":0} /tidb/cdc/default/default/task/position/962197a4-05d7-418c-856b-5c08556d01ee/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9af6bce4-b810-45a5-a132-1b560a622f4a/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:18:12 CST 2024] <<<<<< START cdc server in move_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.move_table.1754717549.out server --log-file /tmp/tidb_cdc_test/move_table/cdc3.log --log-level debug --data-dir /tmp/tidb_cdc_test/move_table/cdc_data3 --cluster-id default --addr 127.0.0.1:8302 * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + echo '' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/cockroachdb/tokenbucket v0.0.0-20230807174530-cc333fc44b06 go: downloading github.com/DataDog/zstd v1.5.5 empty select id, val from test.availability3 where id=1 run task successfully delete_owner_key pass Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... table test.finish_mark not exists for 11-th check, retry later table processor_delay.t36 exists table processor_delay.t37 exists table processor_delay.t38 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 39-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_kill_capture [Fri Apr 26 19:18:14 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.51265128.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_capture.server1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info table mark.finish_mark_3 not exists for 43-th check, retry later * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:14 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb07c11c-5cd1-4e88-875e-135d1af09042 {"id":"cb07c11c-5cd1-4e88-875e-135d1af09042","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20520df4 cb07c11c-5cd1-4e88-875e-135d1af09042 /tidb/cdc/default/default/upstream/7362133493580431422 {"id":7362133493580431422,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb07c11c-5cd1-4e88-875e-135d1af09042 {"id":"cb07c11c-5cd1-4e88-875e-135d1af09042","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20520df4 cb07c11c-5cd1-4e88-875e-135d1af09042 /tidb/cdc/default/default/upstream/7362133493580431422 {"id":7362133493580431422,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cb07c11c-5cd1-4e88-875e-135d1af09042 {"id":"cb07c11c-5cd1-4e88-875e-135d1af09042","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20520df4 cb07c11c-5cd1-4e88-875e-135d1af09042 /tidb/cdc/default/default/upstream/7362133493580431422 {"id":7362133493580431422,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table ddl_only_block_related_table.finish_mark exists wait process 7060 exit for 1-th time... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table test.finish_mark not exists for 12-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:15 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {UpstreamID:7362133410276671937 Namespace:default ID:21eb0bcd-3128-4bef-8bae-9c18b3b46ddc SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:07.415727236 +0800 CST StartTs:449348967303217153 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002fa5320 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348970055467012} {CheckpointTs:449348968141815860 MinTableBarrierTs:449348971942903819 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7fcc6144-db29-4c00-a549-759c987684b2 {"id":"7fcc6144-db29-4c00-a549-759c987684b2","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9af6bce4-b810-45a5-a132-1b560a622f4a {"id":"9af6bce4-b810-45a5-a132-1b560a622f4a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb5b 9af6bce4-b810-45a5-a132-1b560a622f4a /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb7a 7fcc6144-db29-4c00-a549-759c987684b2 /tidb/cdc/default/default/changefeed/info/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"upstream-id":7362133410276671937,"namespace":"default","changefeed-id":"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:07.415727236+08:00","start-ts":449348967303217153,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348970055467012} /tidb/cdc/default/default/changefeed/status/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":449348968141815860,"min-table-barrier-ts":449348971942903819,"admin-job-type":0} /tidb/cdc/default/default/task/position/7fcc6144-db29-4c00-a549-759c987684b2/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/962197a4-05d7-418c-856b-5c08556d01ee/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9af6bce4-b810-45a5-a132-1b560a622f4a/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {UpstreamID:7362133410276671937 Namespace:default ID:21eb0bcd-3128-4bef-8bae-9c18b3b46ddc SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:07.415727236 +0800 CST StartTs:449348967303217153 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002fa5320 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348970055467012} {CheckpointTs:449348968141815860 MinTableBarrierTs:449348971942903819 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7fcc6144-db29-4c00-a549-759c987684b2 {"id":"7fcc6144-db29-4c00-a549-759c987684b2","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9af6bce4-b810-45a5-a132-1b560a622f4a {"id":"9af6bce4-b810-45a5-a132-1b560a622f4a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb5b 9af6bce4-b810-45a5-a132-1b560a622f4a /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb7a 7fcc6144-db29-4c00-a549-759c987684b2 /tidb/cdc/default/default/changefeed/info/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"upstream-id":7362133410276671937,"namespace":"default","changefeed-id":"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:07.415727236+08:00","start-ts":449348967303217153,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348970055467012} /tidb/cdc/default/default/changefeed/status/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":449348968141815860,"min-table-barrier-ts":449348971942903819,"admin-job-type":0} /tidb/cdc/default/default/task/position/7fcc6144-db29-4c00-a549-759c987684b2/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/962197a4-05d7-418c-856b-5c08556d01ee/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9af6bce4-b810-45a5-a132-1b560a622f4a/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** processors info ***: changefeedID: default/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {UpstreamID:7362133410276671937 Namespace:default ID:21eb0bcd-3128-4bef-8bae-9c18b3b46ddc SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:07.415727236 +0800 CST StartTs:449348967303217153 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002fa5320 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348970055467012} {CheckpointTs:449348968141815860 MinTableBarrierTs:449348971942903819 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7fcc6144-db29-4c00-a549-759c987684b2 {"id":"7fcc6144-db29-4c00-a549-759c987684b2","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/962197a4-05d7-418c-856b-5c08556d01ee {"id":"962197a4-05d7-418c-856b-5c08556d01ee","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9af6bce4-b810-45a5-a132-1b560a622f4a {"id":"9af6bce4-b810-45a5-a132-1b560a622f4a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb17 962197a4-05d7-418c-856b-5c08556d01ee /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb5b 9af6bce4-b810-45a5-a132-1b560a622f4a /tidb/cdc/default/__cdc_meta__/owner/22318f1a2010cb7a 7fcc6144-db29-4c00-a549-759c987684b2 /tidb/cdc/default/default/changefeed/info/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"upstream-id":7362133410276671937,"namespace":"default","changefeed-id":"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:07.415727236+08:00","start-ts":449348967303217153,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348970055467012} /tidb/cdc/default/default/changefeed/status/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":449348968141815860,"min-table-barrier-ts":449348971942903819,"admin-job-type":0} /tidb/cdc/default/default/task/position/7fcc6144-db29-4c00-a549-759c987684b2/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/962197a4-05d7-418c-856b-5c08556d01ee/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9af6bce4-b810-45a5-a132-1b560a622f4a/21eb0bcd-3128-4bef-8bae-9c18b3b46ddc {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133410276671937 {"id":7362133410276671937,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x wait process 7060 exit for 2-th time... table processor_delay.t38 exists table processor_delay.t39 not exists for 1-th check, retry later table move_table.usertable exists go: downloading github.com/IBM/sarama v1.41.2 go: downloading github.com/xdg/scram v1.0.5 go: downloading github.com/benbjohnson/clock v1.3.5 go: downloading go.etcd.io/etcd/server/v3 v3.5.12 go: downloading github.com/tinylib/msgp v1.1.6 go: downloading github.com/go-mysql-org/go-mysql v1.7.1-0.20230619063055-fd67d94318fd go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5 go: downloading github.com/KimMachineGun/automemlimit v0.2.4 go: downloading github.com/apache/pulsar-client-go v0.11.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.19.1 go: downloading github.com/pierrec/lz4/v4 v4.1.18 go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2 go: downloading github.com/stretchr/objx v0.5.2 table test.finish_mark not exists for 40-th check, retry later wait process 7060 exit for 3-th time... go: downloading github.com/containerd/cgroups v1.0.4 go: downloading github.com/xdg/stringprep v1.0.3 go: downloading github.com/philhofer/fwd v1.1.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/cilium/ebpf v0.4.0 go: downloading github.com/godbus/dbus/v5 v5.0.4 go: downloading github.com/opencontainers/runtime-spec v1.0.2 go: downloading github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 go: downloading github.com/siddontang/go-log v0.0.0-20180807004314-8d05993dda07 go: downloading github.com/aws/smithy-go v1.13.5 go: downloading github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688260180016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:9481, start at 2024-04-26 19:18:14.831203229 +0800 CST m=+5.102488953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:14.837 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:14.840 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:14.840 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688260180016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:9481, start at 2024-04-26 19:18:14.831203229 +0800 CST m=+5.102488953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:14.837 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:14.840 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:14.840 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6882632c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:9558, start at 2024-04-26 19:18:15.006573039 +0800 CST m=+5.228101982 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:15.013 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:14.987 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:14.987 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_3 not exists for 44-th check, retry later go: downloading github.com/hashicorp/go-multierror v1.1.1 go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 go: downloading github.com/eapache/queue v1.1.0 go: downloading github.com/jcmturner/gokrb5/v8 v8.4.4 go: downloading github.com/eapache/go-resiliency v1.4.0 go: downloading github.com/jcmturner/gofork v1.7.6 go: downloading github.com/hashicorp/errwrap v1.0.0 go: downloading github.com/pierrec/lz4 v2.6.1+incompatible go: downloading github.com/linkedin/goavro/v2 v2.11.1 go: downloading github.com/bits-and-blooms/bitset v1.4.0 go: downloading github.com/AthenZ/athenz v1.10.39 go: downloading golang.org/x/mod v0.17.0 go: downloading github.com/99designs/keyring v1.2.1 go: downloading github.com/spaolacci/murmur3 v1.1.0 go: downloading github.com/mtibben/percent v0.2.1 go: downloading github.com/dvsekhvalnov/jose2go v1.5.0 go: downloading github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c table test.table1 not exists for 1-th check, retry later /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7060) - No such process wait process 7060 exit for 4-th time... process 7060 already exit [Fri Apr 26 19:18:16 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ExecuteNotDone=return(true)' + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.71607162.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0 go: downloading go.opentelemetry.io/otel/sdk v1.22.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 go: downloading github.com/jcmturner/rpc/v2 v2.0.3 go: downloading github.com/jcmturner/aescts/v2 v2.0.0 go: downloading github.com/hashicorp/go-uuid v1.0.3 go: downloading go.etcd.io/bbolt v1.3.8 go: downloading go.etcd.io/etcd/pkg/v3 v3.5.12 go: downloading go.etcd.io/etcd/raft/v3 v3.5.12 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading go.etcd.io/etcd/client/v2 v2.305.12 go: downloading github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.22.0 go: downloading github.com/jonboulle/clockwork v0.4.0 go: downloading go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 go: downloading github.com/gorilla/websocket v1.5.1 go: downloading go.opentelemetry.io/proto/otlp v1.1.0 go: downloading github.com/cenkalti/backoff/v4 v4.2.1 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) go: downloading github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 table test.finish_mark not exists for 13-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table processor_delay.t39 exists table processor_delay.t40 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:17 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ea3320 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348972564185092 MinTableBarrierTs:449348972564185093 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348972564185092, checkpointTs: 449348972564185092, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348972564185092, checkpointTs: 449348972564185092, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348972564185093, checkpointTs: 449348972564185093, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e8a98e2-8f00-4565-9c7b-33d416239db5 {"id":"2e8a98e2-8f00-4565-9c7b-33d416239db5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2d9d 2e8a98e2-8f00-4565-9c7b-33d416239db5 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348972564185092,"min-table-barrier-ts":449348972564185093,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e8a98e2-8f00-4565-9c7b-33d416239db5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ea3320 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348972564185092 MinTableBarrierTs:449348972564185093 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348972564185092, checkpointTs: 449348972564185092, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348972564185092, checkpointTs: 449348972564185092, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348972564185093, checkpointTs: 449348972564185093, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e8a98e2-8f00-4565-9c7b-33d416239db5 {"id":"2e8a98e2-8f00-4565-9c7b-33d416239db5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2d9d 2e8a98e2-8f00-4565-9c7b-33d416239db5 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348972564185092,"min-table-barrier-ts":449348972564185093,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e8a98e2-8f00-4565-9c7b-33d416239db5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003ea3320 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348972564185092 MinTableBarrierTs:449348972564185093 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348972564185092, checkpointTs: 449348972564185092, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348972564185092, checkpointTs: 449348972564185092, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348972564185093, checkpointTs: 449348972564185093, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e8a98e2-8f00-4565-9c7b-33d416239db5 {"id":"2e8a98e2-8f00-4565-9c7b-33d416239db5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2d9d 2e8a98e2-8f00-4565-9c7b-33d416239db5 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348972564185092,"min-table-barrier-ts":449348972564185093,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e8a98e2-8f00-4565-9c7b-33d416239db5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' table test.finish_mark not exists for 41-th check, retry later go: downloading github.com/googleapis/enterprise-certificate-proxy v0.3.2 go: downloading go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 go: downloading github.com/google/s2a-go v0.1.7 go: downloading github.com/felixge/httpsnoop v1.0.4 [Fri Apr 26 19:18:18 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_file case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file.1079610798.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table test.table1 exists table test.table2 not exists for 1-th check, retry later go: downloading github.com/jmespath/go-jmespath v0.4.0 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 45-th check, retry later "is-owner": true, run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688294100013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:6810, start at 2024-04-26 19:18:18.152764462 +0800 CST m=+5.148091113 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:18.159 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:18.116 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:18.116 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688294100013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:6810, start at 2024-04-26 19:18:18.152764462 +0800 CST m=+5.148091113 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:18.159 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:18.116 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:18.116 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68829584000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:6904, start at 2024-04-26 19:18:18.218838824 +0800 CST m=+5.163176162 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:18.225 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:18.209 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:18.209 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/kill_owner_with_ddl/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.finish_mark not exists for 14-th check, retry later go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/json-iterator/go v1.1.12 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:19 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133426535730208 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:04.112622636 +0800 CST StartTs:449348969158148100 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037a2fc0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348969197207553} {CheckpointTs:449348973011402757 MinTableBarrierTs:449348973011402757 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/96cf5f9c-39c3-4505-8cac-206587f076b0 {"id":"96cf5f9c-39c3-4505-8cac-206587f076b0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2018528e 96cf5f9c-39c3-4505-8cac-206587f076b0 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133426535730208,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:18:04.112622636+08:00","start-ts":449348969158148100,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348969197207553} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348973011402757,"min-table-barrier-ts":449348973011402757,"admin-job-type":0} /tidb/cdc/default/default/task/position/96cf5f9c-39c3-4505-8cac-206587f076b0/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133426535730208 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:04.112622636 +0800 CST StartTs:449348969158148100 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037a2fc0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348969197207553} {CheckpointTs:449348973011402757 MinTableBarrierTs:449348973011402757 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/96cf5f9c-39c3-4505-8cac-206587f076b0 {"id":"96cf5f9c-39c3-4505-8cac-206587f076b0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2018528e 96cf5f9c-39c3-4505-8cac-206587f076b0 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133426535730208,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:18:04.112622636+08:00","start-ts":449348969158148100,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348969197207553} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348973011402757,"min-table-barrier-ts":449348973011402757,"admin-job-type":0} /tidb/cdc/default/default/task/position/96cf5f9c-39c3-4505-8cac-206587f076b0/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133426535730208 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:04.112622636 +0800 CST StartTs:449348969158148100 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037a2fc0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348969197207553} {CheckpointTs:449348973011402757 MinTableBarrierTs:449348973011402757 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348973011402757, checkpointTs: 449348973011402757, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/96cf5f9c-39c3-4505-8cac-206587f076b0 {"id":"96cf5f9c-39c3-4505-8cac-206587f076b0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2018528e 96cf5f9c-39c3-4505-8cac-206587f076b0 /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133426535730208,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:18:04.112622636+08:00","start-ts":449348969158148100,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348969197207553} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348973011402757,"min-table-barrier-ts":449348973011402757,"admin-job-type":0} /tidb/cdc/default/default/task/position/96cf5f9c-39c3-4505-8cac-206587f076b0/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"p+ grep -q 'etcd info' d-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table processor_delay.t40 exists table processor_delay.t41 not exists for 1-th check, retry later table mark.finish_mark_3 not exists for 46-th check, retry later table test.table2 exists go: downloading github.com/ardielle/ardielle-go v1.5.2 owner pid: 5131 owner id 2e8a98e2-8f00-4565-9c7b-33d416239db5 table test.availability1 exists nonempty select id, val from test.availability1 where id=1 and val=1 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table test.finish_mark exists check diff successfully check diff failed 1-th time, retry later check diff failed 1-th time, retry later wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:21 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb {"id":"07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a207059f1 07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb /tidb/cdc/default/default/upstream/7362133522636522512 {"id":7362133522636522512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb {"id":"07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a207059f1 07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb /tidb/cdc/default/default/upstream/7362133522636522512 {"id":7362133522636522512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb {"id":"07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a207059f1 07c2e9fb-4118-4eea-b45d-a2d0fa74b7cb /tidb/cdc/default/default/upstream/7362133522636522512 {"id":7362133522636522512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x wait process cdc.test exit for 2-th time... table test.finish_mark not exists for 15-th check, retry later [Fri Apr 26 19:18:21 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.81818183.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:18:21 CST 2024] <<<<<< run test case tidb_mysql_test success! >>>>>> table processor_delay.t41 exists table processor_delay.t42 not exists for 1-th check, retry later nonempty select id, val from test.availability1 where id=1 and val=1 run task successfully [Fri Apr 26 19:18:22 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.52595261.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_kill_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_kill_capture.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ***************** properties ***************** "workload"="core" "readallfields"="true" "threadcount"="10" "mysql.user"="root" "scanproportion"="0" "requestdistribution"="uniform" "fieldcount"="100" "updateproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" "recordcount"="5000" "insertproportion"="0" "dotransactions"="false" "mysql.port"="4000" "readproportion"="0" "mysql.db"="consistent_replicate_storage_file" ********************************************** table mark.finish_mark_3 not exists for 47-th check, retry later check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6882b1680013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:13249, start at 2024-04-26 19:18:20.022815272 +0800 CST m=+5.297117071 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:20.029 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:19.994 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:19.994 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6882b1680013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:13249, start at 2024-04-26 19:18:20.022815272 +0800 CST m=+5.297117071 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:20.029 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:19.994 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:19.994 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6882b2e00009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:13319, start at 2024-04-26 19:18:20.097298348 +0800 CST m=+5.296643386 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:20.104 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:20.088 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:20.088 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/savepoint/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/savepoint/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/savepoint/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff successfully check_ts_not_forward ddl-only-block-related-table table test.finish_mark not exists for 16-th check, retry later table processor_delay.t42 exists table processor_delay.t43 not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:24 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4efc7df3-b65f-453a-97d3-a004c5668b20 {"id":"4efc7df3-b65f-453a-97d3-a004c5668b20","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208302f2 4efc7df3-b65f-453a-97d3-a004c5668b20 /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4efc7df3-b65f-453a-97d3-a004c5668b20 {"id":"4efc7df3-b65f-453a-97d3-a004c5668b20","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208302f2 4efc7df3-b65f-453a-97d3-a004c5668b20 /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4efc7df3-b65f-453a-97d3-a004c5668b20 {"id":"4efc7df3-b65f-453a-97d3-a004c5668b20","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208302f2 4efc7df3-b65f-453a-97d3-a004c5668b20 /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table mark.finish_mark_3 not exists for 48-th check, retry later Create changefeed successfully! ID: 8d12ecb2-6a09-4d09-9b14-4d59459739c9 Info: {"upstream_id":7362133534577049106,"namespace":"default","id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:18:24.635874324+08:00","start_ts":449348974547304450,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348974547304450,"checkpoint_ts":449348974547304450,"checkpoint_time":"2024-04-26 19:18:24.517"} table kill_owner_with_ddl.t1 not exists for 1-th check, retry later Run finished, takes 2.328766604s INSERT - Takes(s): 2.3, Count: 5000, OPS: 2155.7, Avg(us): 4206, Min(us): 1344, Max(us): 169497, 95th(us): 4000, 99th(us): 56000 table test.finish_mark exists check diff successfully Resume changefeed with checkpointTs3 449348974407319556 table test.table5 not exists for 1-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14641.out cli tso query --pd=http://127.0.0.1:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:25 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035e6000 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348974661599235 MinTableBarrierTs:449348974661599236 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e8a98e2-8f00-4565-9c7b-33d416239db5 {"id":"2e8a98e2-8f00-4565-9c7b-33d416239db5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9dc98435-f715-448e-b137-07335da1a0a5 {"id":"9dc98435-f715-448e-b137-07335da1a0a5","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2d9d 2e8a98e2-8f00-4565-9c7b-33d416239db5 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2dde 9dc98435-f715-448e-b137-07335da1a0a5 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348974661599235,"min-table-barrier-ts":449348974661599236,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e8a98e2-8f00-4565-9c7b-33d416239db5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9dc98435-f715-448e-b137-07335da1a0a5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035e6000 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348974661599235 MinTableBarrierTs:449348974661599236 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e8a98e2-8f00-4565-9c7b-33d416239db5 {"id":"2e8a98e2-8f00-4565-9c7b-33d416239db5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9dc98435-f715-448e-b137-07335da1a0a5 {"id":"9dc98435-f715-448e-b137-07335da1a0a5","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2d9d 2e8a98e2-8f00-4565-9c7b-33d416239db5 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2dde 9dc98435-f715-448e-b137-07335da1a0a5 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348974661599235,"min-table-barrier-ts":449348974661599236,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e8a98e2-8f00-4565-9c7b-33d416239db5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9dc98435-f715-448e-b137-07335da1a0a5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 + grep -q 'failed to get info:' {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0035e6000 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348974661599235 MinTableBarrierTs:449348974661599236 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2e8a98e2-8f00-4565-9c7b-33d416239db5 {"id":"2e8a98e2-8f00-4565-9c7b-33d416239db5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/9dc98435-f715-448e-b137-07335da1a0a5 {"id":"9dc98435-f715-448e-b137-07335da1a0a5","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 2 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2d9d 2e8a98e2-8f00-4565-9c7b-33d416239db5 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2dde 9dc98435-f715-448e-b137-07335da1a0a5 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348974661599235,"min-table-barrier-ts":449348974661599236,"admin-job-type":0} /tidb/cdc/default/default/task/position/2e8a98e2-8f00-4565-9c7b-33d416239db5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/9dc98435-f715-448e-b137-07335da1a0a5/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 + grep -q 'etcd info' {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep -v "2e8a98e2-8f00-4565-9c7b-33d416239db5" | grep id wait process cdc.test exit for 1-th time... table processor_delay.t43 exists table processor_delay.t44 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... table mark.finish_mark_3 not exists for 49-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:18:26 CST 2024] <<<<<< run test case many_pk_or_uk success! >>>>>> + set +x + tso='449348974777204737 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348974777204737 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:18:26 CST 2024] <<<<<< START cdc server in savepoint case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.1467514677.out server --log-file /tmp/tidb_cdc_test/savepoint/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/savepoint/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 "id": "9dc98435-f715-448e-b137-07335da1a0a5", "cluster-id": "default" "cluster-id": "default" run task successfully table consistent_replicate_storage_file.usertable exists table consistent_replicate_storage_file.check1 not exists for 1-th check, retry later table kill_owner_with_ddl.t1 exists wait process 8186 exit for 1-th time... table processor_delay.t44 exists table processor_delay.t45 not exists for 1-th check, retry later wait process 8186 exit for 2-th time... table test.table5 not exists for 2-th check, retry later nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later table mark.finish_mark_3 not exists for 50-th check, retry later /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8186) - No such process wait process 8186 exit for 3-th time... process 8186 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 26 19:18:28 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.83398341.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_storage_file.check1 not exists for 2-th check, retry later table processor_delay.t45 exists table processor_delay.t46 not exists for 1-th check, retry later table test.table5 exists table test.table6 not exists for 1-th check, retry later nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later table mark.finish_mark_3 not exists for 51-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:30 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d808c9d0-d064-408f-8d52-d2502597632d {"id":"d808c9d0-d064-408f-8d52-d2502597632d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2081bef7 d808c9d0-d064-408f-8d52-d2502597632d /tidb/cdc/default/default/upstream/7362133534615969657 {"id":7362133534615969657,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d808c9d0-d064-408f-8d52-d2502597632d {"id":"d808c9d0-d064-408f-8d52-d2502597632d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2081bef7 d808c9d0-d064-408f-8d52-d2502597632d /tidb/cdc/default/default/upstream/7362133534615969657 {"id":7362133534615969657,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d808c9d0-d064-408f-8d52-d2502597632d {"id":"d808c9d0-d064-408f-8d52-d2502597632d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2081bef7 d808c9d0-d064-408f-8d52-d2502597632d /tidb/cdc/default/default/upstream/7362133534615969657 {"id":7362133534615969657,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.savepoint.cli.14723.out cli changefeed create --start-ts=449348974777204737 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 7abd9eb3-856e-4b99-892c-8c3a5c254df2 Info: {"upstream_id":7362133534615969657,"namespace":"default","id":"7abd9eb3-856e-4b99-892c-8c3a5c254df2","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:18:30.922550872+08:00","start_ts":449348974777204737,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348974777204737,"checkpoint_ts":449348974777204737,"checkpoint_time":"2024-04-26 19:18:25.394"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } table consistent_replicate_storage_file.check1 not exists for 3-th check, retry later [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:31 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003626cf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976264347655 MinTableBarrierTs:449348976264347655 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976264347655, checkpointTs: 449348976264347655, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3508fcca-1437-4d6e-8d6e-ffa00065e9b2 {"id":"3508fcca-1437-4d6e-8d6e-ffa00065e9b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830353 3508fcca-1437-4d6e-8d6e-ffa00065e9b2 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976264347655,"min-table-barrier-ts":449348976264347655,"admin-job-type":0} /tidb/cdc/default/default/task/position/3508fcca-1437-4d6e-8d6e-ffa00065e9b2/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003626cf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976264347655 MinTableBarrierTs:449348976264347655 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976264347655, checkpointTs: 449348976264347655, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3508fcca-1437-4d6e-8d6e-ffa00065e9b2 {"id":"3508fcca-1437-4d6e-8d6e-ffa00065e9b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830353 3508fcca-1437-4d6e-8d6e-ffa00065e9b2 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976264347655,"min-table-barrier-ts":449348976264347655,"admin-job-type":0} /tidb/cdc/default/default/task/position/3508fcca-1437-4d6e-8d6e-ffa00065e9b2/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003626cf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976264347655 MinTableBarrierTs:449348976264347655 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976264347655, checkpointTs: 449348976264347655, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/3508fcca-1437-4d6e-8d6e-ffa00065e9b2 {"id":"3508fcca-1437-4d6e-8d6e-ffa00065e9b2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830353 3508fcca-1437-4d6e-8d6e-ffa00065e9b2 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976264347655,"min-table-barrier-ts":449348976264347655,"admin-job-type":0} /tidb/cdc/default/default/task/position/3508fcca-1437-4d6e-8d6e-ffa00065e9b2/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully table test.table6 exists check diff failed 1-th time, retry later [Pipeline] // node =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_reentrant/run.sh using Sink-Type: mysql... <<================= ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) The 1 times to try to start tidb cluster... [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table processor_delay.t46 exists table processor_delay.t47 not exists for 1-th check, retry later + set +x table mark.finish_mark_3 not exists for 52-th check, retry later table consistent_replicate_storage_file.check1 not exists for 4-th check, retry later check diff successfully Resume changefeed with checkpointTs1 449348972074237957 table savepoint.finish_mark not exists for 1-th check, retry later table processor_delay.t47 exists table processor_delay.t48 not exists for 1-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_reentrant Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... nonempty select id, val from test.availability1 where id=2 and val=2 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 3-th time, retry later table test.table3 not exists for 1-th check, retry later table mark.finish_mark_3 not exists for 53-th check, retry later table consistent_replicate_storage_file.check1 not exists for 5-th check, retry later Verifying downstream PD is started... table savepoint.finish_mark not exists for 2-th check, retry later table processor_delay.t48 exists table processor_delay.t49 not exists for 1-th check, retry later table test.table3 not exists for 2-th check, retry later table mark.finish_mark_3 not exists for 54-th check, retry later table consistent_replicate_storage_file.check1 not exists for 6-th check, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table savepoint.finish_mark exists check diff successfully table processor_delay.t49 exists table processor_delay.t50 not exists for 1-th check, retry later wait process 8344 exit for 1-th time... wait process 8344 exit for 2-th time... table mark.finish_mark_3 not exists for 55-th check, retry later wait process 8344 exit for 3-th time... wait process cdc.test exit for 1-th time... table test.table3 not exists for 3-th check, retry later table consistent_replicate_storage_file.check1 not exists for 7-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8344) - No such process wait process 8344 exit for 4-th time... process 8344 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 26 19:18:39 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.85208522.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:18:39 CST 2024] <<<<<< run test case savepoint success! >>>>>> table processor_delay.t50 exists table mark.finish_mark_3 not exists for 56-th check, retry later table test.table3 exists table test.table4 not exists for 1-th check, retry later nonempty select id, val from test.availability1 where id=2 and val=2 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table consistent_replicate_storage_file.check1 not exists for 8-th check, retry later check diff failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/capture_session_done_during_task/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_hang_up_capture [Fri Apr 26 19:18:41 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.54345436.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_capture.server1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:42 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034bb950 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4686e862-b289-4ad8-83f5-2e8c6cb3c346 {"id":"4686e862-b289-4ad8-83f5-2e8c6cb3c346","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208303aa 4686e862-b289-4ad8-83f5-2e8c6cb3c346 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/4686e862-b289-4ad8-83f5-2e8c6cb3c346/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034bb950 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4686e862-b289-4ad8-83f5-2e8c6cb3c346 {"id":"4686e862-b289-4ad8-83f5-2e8c6cb3c346","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208303aa 4686e862-b289-4ad8-83f5-2e8c6cb3c346 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/4686e862-b289-4ad8-83f5-2e8c6cb3c346/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 + grep -q 'failed to get info:' {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0034bb950 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4686e862-b289-4ad8-83f5-2e8c6cb3c346 {"id":"4686e862-b289-4ad8-83f5-2e8c6cb3c346","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208303aa 4686e862-b289-4ad8-83f5-2e8c6cb3c346 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/4686e862-b289-4ad8-83f5-2e8c6cb3c346/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully start tidb cluster in /tmp/tidb_cdc_test/capture_session_done_during_task Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table test.table4 exists check diff failed 1-th time, retry later table mark.finish_mark_3 not exists for 57-th check, retry later table consistent_replicate_storage_file.check1 not exists for 9-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_3 not exists for 58-th check, retry later check diff successfully Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check diff successfully table consistent_replicate_storage_file.check1 not exists for 10-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:44 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002bd03f0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348979642073095 MinTableBarrierTs:449348979642073096 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348979642073095, checkpointTs: 449348979642073095, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348979642073096, checkpointTs: 449348979642073096, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348979642073095, checkpointTs: 449348979642073095, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2887b3bd-30af-4d96-8383-f1a1e0eabd4f {"id":"2887b3bd-30af-4d96-8383-f1a1e0eabd4f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e4c 2887b3bd-30af-4d96-8383-f1a1e0eabd4f /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348979642073095,"min-table-barrier-ts":449348979642073096,"admin-job-type":0} /tidb/cdc/default/default/task/position/2887b3bd-30af-4d96-8383-f1a1e0eabd4f/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002bd03f0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348979642073095 MinTableBarrierTs:449348979642073096 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348979642073095, checkpointTs: 449348979642073095, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348979642073096, checkpointTs: 449348979642073096, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348979642073095, checkpointTs: 449348979642073095, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2887b3bd-30af-4d96-8383-f1a1e0eabd4f {"id":"2887b3bd-30af-4d96-8383-f1a1e0eabd4f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e4c 2887b3bd-30af-4d96-8383-f1a1e0eabd4f /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348979642073095,"min-table-barrier-ts":449348979642073096,"admin-job-type":0} /tidb/cdc/default/default/task/position/2887b3bd-30af-4d96-8383-f1a1e0eabd4f/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-p+ grep -q 'failed to get info:' ath":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002bd03f0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348979642073095 MinTableBarrierTs:449348979642073096 AdminJobType:noop} span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348979642073095, checkpointTs: 449348979642073095, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348979642073096, checkpointTs: 449348979642073096, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348979642073095, checkpointTs: 449348979642073095, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2887b3bd-30af-4d96-8383-f1a1e0eabd4f {"id":"2887b3bd-30af-4d96-8383-f1a1e0eabd4f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e4c 2887b3bd-30af-4d96-8383-f1a1e0eabd4f /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348979642073095,"min-table-barrier-ts":449348979642073096,"admin-job-type":0} /tidb/cdc/default/default/task/position/2887b3bd-30af-4d96-8383-f1a1e0eabd4f/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-p+ grep -q 'etcd info' ath":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68842a980018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:6638, start at 2024-04-26 19:18:44.176803621 +0800 CST m=+5.200185326 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:44.184 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:44.184 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:44.184 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68842a980018 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:6638, start at 2024-04-26 19:18:44.176803621 +0800 CST m=+5.200185326 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:44.184 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:44.184 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:44.184 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68842d0c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:6724, start at 2024-04-26 19:18:44.313152693 +0800 CST m=+5.266352231 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:44.320 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:44.291 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:44.291 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_reentrant/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 1-th time... wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:18:46 CST 2024] <<<<<< START cdc server in changefeed_resume_with_checkpoint_ts case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/InjectChangefeedFastFailError=1*return(true)' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_resume_with_checkpoint_ts.84388440.out server --log-file /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_resume_with_checkpoint_ts/cdc_data --cluster-id default --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 3-th time... "is-owner": true, run task successfully table mark.finish_mark_3 exists table mark.finish_mark not exists for 1-th check, retry later [2024/04/26 19:18:43.718 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/04/26 19:18:43.724 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/04/26 19:18:43.725 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=962197a4-05d7-418c-856b-5c08556d01ee] [2024/04/26 19:18:43.726 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/04/26 19:18:43.726 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/04/26 19:18:43.885 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc] [captureID=7fcc6144-db29-4c00-a549-759c987684b2] [processorDetail="{\"table_ids\":[]}"] [2024/04/26 19:18:44.087 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc] [captureID=962197a4-05d7-418c-856b-5c08556d01ee] [processorDetail="{\"table_ids\":[104,106]}"] [2024/04/26 19:18:44.288 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc] [captureID=9af6bce4-b810-45a5-a132-1b560a622f4a] [processorDetail="{\"table_ids\":[]}"] [2024/04/26 19:18:44.288 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"7fcc6144-db29-4c00-a549-759c987684b2\":[],\"962197a4-05d7-418c-856b-5c08556d01ee\":[{\"ID\":104,\"Changefeed\":\"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc\"},{\"ID\":106,\"Changefeed\":\"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc\"}],\"9af6bce4-b810-45a5-a132-1b560a622f4a\":[]}"] [2024/04/26 19:18:44.288 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc&target-cp-id=7fcc6144-db29-4c00-a549-759c987684b2&table-id=104"] [2024/04/26 19:18:44.335 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=104] [2024/04/26 19:18:44.335 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc&target-cp-id=7fcc6144-db29-4c00-a549-759c987684b2&table-id=106"] [2024/04/26 19:18:44.386 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=106] [2024/04/26 19:18:44.386 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=962197a4-05d7-418c-856b-5c08556d01ee] [targetCapture=7fcc6144-db29-4c00-a549-759c987684b2] table move_table.check1 exists check diff successfully table consistent_replicate_storage_file.check1 not exists for 11-th check, retry later cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:18:47 CST 2024] <<<<<< run test case processor_etcd_worker_delay success! >>>>>> Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) skip some DDLs in tidb v4.0.x [Fri Apr 26 19:18:47 CST 2024] <<<<<< START cdc server in ddl_reentrant case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_reentrant.79637965.out server --log-file /tmp/tidb_cdc_test/ddl_reentrant/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_reentrant/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 owner pid: 5439 owner id 2887b3bd-30af-4d96-8383-f1a1e0eabd4f [Fri Apr 26 19:18:47 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.55435545.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_hang_up_capture.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_hang_up_capture.server2 --cluster-id default --addr 127.0.0.1:8301 ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... table consistent_replicate_storage_file.check1 not exists for 12-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:49 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a16cd1c-6491-47a0-8948-22f44511948d {"id":"4a16cd1c-6491-47a0-8948-22f44511948d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20520f27 4a16cd1c-6491-47a0-8948-22f44511948d /tidb/cdc/default/default/changefeed/info/c65e0c31-acec-457f-9777-4ffe69f0f906 {"upstream-id":7362133493580431422,"namespace":"default","changefeed-id":"c65e0c31-acec-457f-9777-4ffe69f0f906","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:15.028105457+08:00","start-ts":449348972074237957,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"failed","error":{"time":"2024-04-26T19:18:46.741487058+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348980371095559} /tidb/cdc/default/default/changefeed/status/c65e0c31-acec-457f-9777-4ffe69f0f906 {"checkpoint-ts":449348974420688920,"min-table-barrier-ts":449348974564605980,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362133493580431422 {"id":7362133493580431422,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a16cd1c-6491-47a0-8948-22f44511948d {"id":"4a16cd1c-6491-47a0-8948-22f44511948d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20520f27 4a16cd1c-6491-47a0-8948-22f44511948d /tidb/cdc/default/default/changefeed/info/c65e0c31-acec-457f-9777-4ffe69f0f906 {"upstream-id":7362133493580431422,"namespace":"default","changefeed-id":"c65e0c31-acec-457f-9777-4ffe69f0f906","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:15.028105457+08:00","start-ts":449348972074237957,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"failed","error":{"time":"2024-04-26T19:18:46.741487058+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348980371095559} /tidb/cdc/default/default/changefeed/status/c65e0c31-acec-457f-9777-4ffe69f0f906 {"checkpoint-ts":449348974420688920,"min-table-barrier-ts":449348974564605980,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362133493580431422 {"id":7362133493580431422,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4a16cd1c-6491-47a0-8948-22f44511948d {"id":"4a16cd1c-6491-47a0-8948-22f44511948d","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20520f27 4a16cd1c-6491-47a0-8948-22f44511948d /tidb/cdc/default/default/changefeed/info/c65e0c31-acec-457f-9777-4ffe69f0f906 {"upstream-id":7362133493580431422,"namespace":"default","changefeed-id":"c65e0c31-acec-457f-9777-4ffe69f0f906","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:15.028105457+08:00","start-ts":449348972074237957,"target-ts":0,"admin-job-type":1,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"failed","error":{"time":"2024-04-26T19:18:46.741487058+08:00","addr":"127.0.0.1:8300","code":"CDC:ErrStartTsBeforeGC","message":"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"},"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348980371095559} /tidb/cdc/default/default/changefeed/status/c65e0c31-acec-457f-9777-4ffe69f0f906 {"checkpoint-ts":449348974420688920,"min-table-barrier-ts":449348974564605980,"admin-job-type":1} /tidb/cdc/default/default/upstream/7362133493580431422 {"id":7362133493580431422,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 c65e0c31-acec-457f-9777-4ffe69f0f906 failed ErrStartTsBeforeGC + endpoints=http://127.0.0.1:2379 + changefeed_id=c65e0c31-acec-457f-9777-4ffe69f0f906 + expected_state=failed + error_msg=ErrStartTsBeforeGC + tls_dir=ErrStartTsBeforeGC + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c c65e0c31-acec-457f-9777-4ffe69f0f906 -s + info='{ "upstream_id": 7362133493580431422, "namespace": "default", "id": "c65e0c31-acec-457f-9777-4ffe69f0f906", "state": "failed", "checkpoint_tso": 449348974420688920, "checkpoint_time": "2024-04-26 19:18:24.034", "error": { "time": "2024-04-26T19:18:46.741487058+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' + echo '{ "upstream_id": 7362133493580431422, "namespace": "default", "id": "c65e0c31-acec-457f-9777-4ffe69f0f906", "state": "failed", "checkpoint_tso": 449348974420688920, "checkpoint_time": "2024-04-26 19:18:24.034", "error": { "time": "2024-04-26T19:18:46.741487058+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } }' { "upstream_id": 7362133493580431422, "namespace": "default", "id": "c65e0c31-acec-457f-9777-4ffe69f0f906", "state": "failed", "checkpoint_tso": 449348974420688920, "checkpoint_time": "2024-04-26 19:18:24.034", "error": { "time": "2024-04-26T19:18:46.741487058+08:00", "addr": "127.0.0.1:8300", "code": "CDC:ErrStartTsBeforeGC", "message": "[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError" } } ++ jq -r .state ++ echo '{' '"upstream_id":' 7362133493580431422, '"namespace":' '"default",' '"id":' '"c65e0c31-acec-457f-9777-4ffe69f0f906",' '"state":' '"failed",' '"checkpoint_tso":' 449348974420688920, '"checkpoint_time":' '"2024-04-26' '19:18:24.034",' '"error":' '{' '"time":' '"2024-04-26T19:18:46.741487058+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' + state=failed + [[ ! failed == \f\a\i\l\e\d ]] ++ echo '{' '"upstream_id":' 7362133493580431422, '"namespace":' '"default",' '"id":' '"c65e0c31-acec-457f-9777-4ffe69f0f906",' '"state":' '"failed",' '"checkpoint_tso":' 449348974420688920, '"checkpoint_time":' '"2024-04-26' '19:18:24.034",' '"error":' '{' '"time":' '"2024-04-26T19:18:46.741487058+08:00",' '"addr":' '"127.0.0.1:8300",' '"code":' '"CDC:ErrStartTsBeforeGC",' '"message":' '"[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError"' '}' '}' ++ jq -r .error.message + message='[CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError' + [[ ! [CDC:ErrStartTsBeforeGC]InjectChangefeedFastFailError =~ ErrStartTsBeforeGC ]] run task successfully =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/synced_status + CDC_BINARY=cdc.test + SINK_TYPE=mysql + CDC_COUNT=3 + DB_COUNT=4 + trap stop_tidb_cluster EXIT + run_normal_case_and_unavailable_pd conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status The 1 times to try to start tidb cluster... wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:50 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b77990f0-5827-4a0b-b2f8-891546becc72 {"id":"b77990f0-5827-4a0b-b2f8-891546becc72","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20e457f4 b77990f0-5827-4a0b-b2f8-891546becc72 /tidb/cdc/default/default/upstream/7362133640998492791 {"id":7362133640998492791,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b77990f0-5827-4a0b-b2f8-891546becc72 {"id":"b77990f0-5827-4a0b-b2f8-891546becc72","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20e457f4 b77990f0-5827-4a0b-b2f8-891546becc72 /tidb/cdc/default/default/upstream/7362133640998492791 {"id":7362133640998492791,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b77990f0-5827-4a0b-b2f8-891546becc72 {"id":"b77990f0-5827-4a0b-b2f8-891546becc72","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20e457f4 b77990f0-5827-4a0b-b2f8-891546becc72 /tidb/cdc/default/default/upstream/7362133640998492791 {"id":7362133640998492791,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:18:50 CST 2024] <<<<<< run test case default_value success! >>>>>> ------------------------------------------ test ddl create database ddl_reentrant, is_reentrant: false restored_sql: CREATE DATABASE `ddl_reentrant` [2024/04/26 19:18:50.765 +08:00] [INFO] [main.go:61] ["table mover started"] [2024/04/26 19:18:50.767 +08:00] [INFO] [main.go:166] ["new cluster initialized"] [2024/04/26 19:18:50.771 +08:00] [DEBUG] [main.go:192] ["retrieved owner ID"] [ownerID=962197a4-05d7-418c-856b-5c08556d01ee] [2024/04/26 19:18:50.772 +08:00] [DEBUG] [main.go:199] ["retrieved owner addr"] [ownerAddr=127.0.0.1:8300] [2024/04/26 19:18:50.773 +08:00] [DEBUG] [main.go:210] ["retrieved changefeeds"] [changefeedsError="json: unsupported type: map[model.ChangeFeedID]*mvccpb.KeyValue"] [2024/04/26 19:18:50.936 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc] [captureID=7fcc6144-db29-4c00-a549-759c987684b2] [processorDetail="{\"table_ids\":[106]}"] [2024/04/26 19:18:51.135 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc] [captureID=962197a4-05d7-418c-856b-5c08556d01ee] [processorDetail="{\"table_ids\":[108]}"] check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:50 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003300b40 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348981215199244 MinTableBarrierTs:449348981215199245 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2887b3bd-30af-4d96-8383-f1a1e0eabd4f {"id":"2887b3bd-30af-4d96-8383-f1a1e0eabd4f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/e4427966-e1b5-4570-9cf8-43bc87b338df {"id":"e4427966-e1b5-4570-9cf8-43bc87b338df","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e4c 2887b3bd-30af-4d96-8383-f1a1e0eabd4f /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e84 e4427966-e1b5-4570-9cf8-43bc87b338df /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348981215199244,"min-table-barrier-ts":449348981215199245,"admin-job-type":0} /tidb/cdc/default/default/task/position/2887b3bd-30af-4d96-8383-f1a1e0eabd4f/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e4427966-e1b5-4570-9cf8-43bc87b338df/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003300b40 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348981215199244 MinTableBarrierTs:449348981215199245 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2887b3bd-30af-4d96-8383-f1a1e0eabd4f {"id":"2887b3bd-30af-4d96-8383-f1a1e0eabd4f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/e4427966-e1b5-4570-9cf8-43bc87b338df {"id":"e4427966-e1b5-4570-9cf8-43bc87b338df","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e4c 2887b3bd-30af-4d96-8383-f1a1e0eabd4f /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e84 e4427966-e1b5-4570-9cf8-43bc87b338df /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348981215199244,"min-table-barrier-ts":449348981215199245,"admin-job-type":0} /tidb/cdc/default/default/task/position/2887b3bd-30af-4d96-8383-f1a1e0eabd4f/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e4427966-e1b5-4570-9cf8-43bc87b338df/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003300b40 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348981215199244 MinTableBarrierTs:449348981215199245 AdminJobType:noop} *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2887b3bd-30af-4d96-8383-f1a1e0eabd4f {"id":"2887b3bd-30af-4d96-8383-f1a1e0eabd4f","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/e4427966-e1b5-4570-9cf8-43bc87b338df {"id":"e4427966-e1b5-4570-9cf8-43bc87b338df","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 3 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e4c 2887b3bd-30af-4d96-8383-f1a1e0eabd4f /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2e84 e4427966-e1b5-4570-9cf8-43bc87b338df /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348981215199244,"min-table-barrier-ts":449348981215199245,"admin-job-type":0} /tidb/cdc/default/default/task/position/2887b3bd-30af-4d96-8383-f1a1e0eabd4f/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/task/position/e4427966-e1b5-4570-9cf8-43bc87b338df/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep -v "2887b3bd-30af-4d96-8383-f1a1e0eabd4f" | grep id [2024/04/26 19:18:51.336 +08:00] [DEBUG] [main.go:229] ["retrieved processor details"] [changefeed=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc] [captureID=9af6bce4-b810-45a5-a132-1b560a622f4a] [processorDetail="{\"table_ids\":[]}"] [2024/04/26 19:18:51.336 +08:00] [INFO] [main.go:75] ["task status"] [status="{\"7fcc6144-db29-4c00-a549-759c987684b2\":[{\"ID\":106,\"Changefeed\":\"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc\"}],\"962197a4-05d7-418c-856b-5c08556d01ee\":[{\"ID\":108,\"Changefeed\":\"21eb0bcd-3128-4bef-8bae-9c18b3b46ddc\"}],\"9af6bce4-b810-45a5-a132-1b560a622f4a\":[]}"] [2024/04/26 19:18:51.336 +08:00] [DEBUG] [main.go:288] ["preparing HTTP API call to owner"] [formStr="cf-id=21eb0bcd-3128-4bef-8bae-9c18b3b46ddc&target-cp-id=9af6bce4-b810-45a5-a132-1b560a622f4a&table-id=106"] [2024/04/26 19:18:51.388 +08:00] [INFO] [main.go:180] ["moved table successful"] [tableID=106] [2024/04/26 19:18:51.388 +08:00] [INFO] [main.go:114] ["all tables are moved"] [sourceCapture=7fcc6144-db29-4c00-a549-759c987684b2] [targetCapture=9af6bce4-b810-45a5-a132-1b560a622f4a] table consistent_replicate_storage_file.check1 not exists for 13-th check, retry later check diff successfully check_changefeed_state http://127.0.0.1:2379 c65e0c31-acec-457f-9777-4ffe69f0f906 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=c65e0c31-acec-457f-9777-4ffe69f0f906 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c c65e0c31-acec-457f-9777-4ffe69f0f906 -s table move_table.check2 not exists for 1-th check, retry later + info='{ "upstream_id": 7362133493580431422, "namespace": "default", "id": "c65e0c31-acec-457f-9777-4ffe69f0f906", "state": "normal", "checkpoint_tso": 449348981590065154, "checkpoint_time": "2024-04-26 19:18:51.383", "error": null }' + echo '{ "upstream_id": 7362133493580431422, "namespace": "default", "id": "c65e0c31-acec-457f-9777-4ffe69f0f906", "state": "normal", "checkpoint_tso": 449348981590065154, "checkpoint_time": "2024-04-26 19:18:51.383", "error": null }' { "upstream_id": 7362133493580431422, "namespace": "default", "id": "c65e0c31-acec-457f-9777-4ffe69f0f906", "state": "normal", "checkpoint_tso": 449348981590065154, "checkpoint_time": "2024-04-26 19:18:51.383", "error": null } ++ echo '{' '"upstream_id":' 7362133493580431422, '"namespace":' '"default",' '"id":' '"c65e0c31-acec-457f-9777-4ffe69f0f906",' '"state":' '"normal",' '"checkpoint_tso":' 449348981590065154, '"checkpoint_time":' '"2024-04-26' '19:18:51.383",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ echo '{' '"upstream_id":' 7362133493580431422, '"namespace":' '"default",' '"id":' '"c65e0c31-acec-457f-9777-4ffe69f0f906",' '"state":' '"normal",' '"checkpoint_tso":' 449348981590065154, '"checkpoint_time":' '"2024-04-26' '19:18:51.383",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully "cluster-id": "default" "id": "e4427966-e1b5-4570-9cf8-43bc87b338df", "cluster-id": "default" run task successfully wait process 8525 exit for 1-th time... wait process 8525 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8525) - No such process wait process 8525 exit for 3-th time... process 8525 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 26 19:18:51 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.86718673.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:18:51.494 +08:00] [INFO] [case.go:115] ["sync updatePKUK take: 12.497955047s"] table consistent_replicate_storage_file.check1 not exists for 14-th check, retry later nonempty select id, val from test.availability1 where id=3 and val=3 changefeed is working normally rts: 449348981354659840->449348981760720900 checkpoint: 449348981354659840->449348981760720900 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/04/26 19:18:52.253 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE DATABASE `ddl_reentrant`"] [duration=80.695392ms] [namespace=default] [changefeed=94572c11-4be2-48f9-adbc-de25eb5fa07c] run task successfully resume changefeed 94572c11-4be2-48f9-adbc-de25eb5fa07c from 449348981393981468 check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c table move_table.check2 exists table test1.table1 not exists for 1-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check diff successfully wait process cdc.test exit for 1-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6884bf680012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:20287, start at 2024-04-26 19:18:53.682219529 +0800 CST m=+6.385234674 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:53.690 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:53.658 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:53.658 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... table consistent_replicate_storage_file.check1 not exists for 15-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:18:55 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002d46a20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9e47b5c-8b8a-4103-a955-38be8234e785 {"id":"a9e47b5c-8b8a-4103-a955-38be8234e785","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208303f3 a9e47b5c-8b8a-4103-a955-38be8234e785 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9e47b5c-8b8a-4103-a955-38be8234e785/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002d46a20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9e47b5c-8b8a-4103-a955-38be8234e785 {"id":"a9e47b5c-8b8a-4103-a955-38be8234e785","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208303f3 a9e47b5c-8b8a-4103-a955-38be8234e785 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9e47b5c-8b8a-4103-a955-38be8234e785/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 + grep -q 'failed to get info:' {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002d46a20 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/a9e47b5c-8b8a-4103-a955-38be8234e785 {"id":"a9e47b5c-8b8a-4103-a955-38be8234e785","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a208303f3 a9e47b5c-8b8a-4103-a955-38be8234e785 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/a9e47b5c-8b8a-4103-a955-38be8234e785/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully table test1.table1 exists table test1.table2 not exists for 1-th check, retry later nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 2-th time, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:18:56 CST 2024] <<<<<< run test case move_table success! >>>>>> changefeed is working normally rts: 449348982023127040->449348982547152900 checkpoint: 449348981393981440->449348982547152900 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2024/04/26 19:18:54.179 +08:00] [INFO] [mysql_ddl_sink.go:119] ["Execute DDL failed, but error can be ignored"] [startTs=449348981393981465] [ddl="CREATE DATABASE `ddl_reentrant`"] [namespace=default] [changefeed=94572c11-4be2-48f9-adbc-de25eb5fa07c] [error="Error 1007 (HY000): Can't create database 'ddl_reentrant'; database exists"] run task successfully ------------------------------------------ test ddl create table ddl_reentrant.t3 (a int primary key, b int) partition by range(a) (partition p0 values less than (1000), partition p1 values less than (2000)), is_reentrant: false restored_sql: CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000)) check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6884bf680012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:20287, start at 2024-04-26 19:18:53.682219529 +0800 CST m=+6.385234674 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:53.690 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:53.658 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:53.658 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6884ad500006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:20364, start at 2024-04-26 19:18:52.507651505 +0800 CST m=+5.158559956 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:20:52.518 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:18:52.500 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:08:52.500 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/capture_session_done_during_task/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table consistent_replicate_storage_file.check1 not exists for 16-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table test1.table2 exists check diff successfully run task failed 1-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... wait process cdc.test exit for 1-th time... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.cli.21712.out cli tso query --pd=http://127.0.0.1:2379 Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/sink_hang/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table consistent_replicate_storage_file.check1 not exists for 17-th check, retry later wait process cdc.test exit for 2-th time... nonempty select id, val from test.availability1 where id=3 and val=3 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 3-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:00 CST 2024] <<<<<< run test case changefeed_resume_with_checkpoint_ts success! >>>>>> table big_txn.finish_mark_1 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:18:51 CST 2024] <<<<<< run test case big_txn success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/simple/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_finish/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... + set +x + tso='449348983680401411 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348983680401411 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:19:00 CST 2024] <<<<<< START cdc server in capture_session_done_during_task case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/processorManagerHandleNewChangefeedDelay=sleep(2000)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.capture_session_done_during_task.2175421756.out server --log-file /tmp/tidb_cdc_test/capture_session_done_during_task/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/capture_session_done_during_task/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file.check1 not exists for 18-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/sink_hang Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... changefeed is working normally rts: 449348983858135040->449348984120016900 checkpoint: 449348983858135040->449348984120016900 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/04/26 19:18:56.208 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000))"] [duration=83.956507ms] [namespace=default] [changefeed=94572c11-4be2-48f9-adbc-de25eb5fa07c] run task successfully resume changefeed 94572c11-4be2-48f9-adbc-de25eb5fa07c from 449348982717808674 check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c start tidb cluster in /tmp/tidb_cdc_test/simple Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table consistent_replicate_storage_file.check1 not exists for 19-th check, retry later wait process 8676 exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 8676 exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/changefeed_finish Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8676) - No such process wait process 8676 exit for 3-th time... process 8676 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 26 19:19:04 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true);github.com/pingcap/tiflow/cdc/capture/ownerFlushIntervalInject=return(10)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.88218823.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 run task failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:05 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b9ee9e0f-2667-4c9c-bdbc-92f673f18107 {"id":"b9ee9e0f-2667-4c9c-bdbc-92f673f18107","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21050f0b b9ee9e0f-2667-4c9c-bdbc-92f673f18107 /tidb/cdc/default/default/upstream/7362133685306064310 {"id":7362133685306064310,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b9ee9e0f-2667-4c9c-bdbc-92f673f18107 {"id":"b9ee9e0f-2667-4c9c-bdbc-92f673f18107","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21050f0b b9ee9e0f-2667-4c9c-bdbc-92f673f18107 /tidb/cdc/default/default/upstream/7362133685306064310 {"id":7362133685306064310,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b9ee9e0f-2667-4c9c-bdbc-92f673f18107 {"id":"b9ee9e0f-2667-4c9c-bdbc-92f673f18107","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21050f0b b9ee9e0f-2667-4c9c-bdbc-92f673f18107 /tidb/cdc/default/default/upstream/7362133685306064310 {"id":7362133685306064310,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table consistent_replicate_storage_file.check1 not exists for 20-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688564080017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:15769, start at 2024-04-26 19:19:04.240272327 +0800 CST m=+5.102503570 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:04.246 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:04.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:04.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688564080017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:15769, start at 2024-04-26 19:19:04.240272327 +0800 CST m=+5.102503570 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:04.246 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:04.245 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:04.245 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688566440013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:15856, start at 2024-04-26 19:19:04.356830408 +0800 CST m=+5.135588771 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:04.363 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:04.337 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:04.337 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release nonempty select id, val from test.availability1 where id=3 and val=3 run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c lease 22318f1a21050f0b revoked table consistent_replicate_storage_file.check1 not exists for 21-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit run test case test_expire_capture [Fri Apr 26 19:19:07 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.57355737.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_expire_capture.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_expire_capture.server1 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.17120.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:07 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002ecd440 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66eb1d9c-8470-4a47-b906-e521b4822787 {"id":"66eb1d9c-8470-4a47-b906-e521b4822787","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830431 66eb1d9c-8470-4a47-b906-e521b4822787 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/66eb1d9c-8470-4a47-b906-e521b4822787/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002ecd440 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66eb1d9c-8470-4a47-b906-e521b4822787 {"id":"66eb1d9c-8470-4a47-b906-e521b4822787","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830431 66eb1d9c-8470-4a47-b906-e521b4822787 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/66eb1d9c-8470-4a47-b906-e521b4822787/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 + grep -q 'failed to get info:' {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002ecd440 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976434741281 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66eb1d9c-8470-4a47-b906-e521b4822787 {"id":"66eb1d9c-8470-4a47-b906-e521b4822787","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830431 66eb1d9c-8470-4a47-b906-e521b4822787 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976434741281,"admin-job-type":0} /tidb/cdc/default/default/task/position/66eb1d9c-8470-4a47-b906-e521b4822787/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully table capture_session_done_during_task.t exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/autorandom/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... check diff failed 1-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) changefeed is working normally rts: 449348985444106240->449348985968132100 checkpoint: 449348985444106240->449348985968132100 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2024/04/26 19:19:03.182 +08:00] [INFO] [mysql_ddl_sink.go:119] ["Execute DDL failed, but error can be ignored"] [startTs=449348982717808667] [ddl="CREATE TABLE `ddl_reentrant`.`t3` (`a` INT PRIMARY KEY,`b` INT) PARTITION BY RANGE (`a`) (PARTITION `p0` VALUES LESS THAN (1000),PARTITION `p1` VALUES LESS THAN (2000))"] [namespace=default] [changefeed=94572c11-4be2-48f9-adbc-de25eb5fa07c] [error="Error 1050 (42S01): Table 'ddl_reentrant.t3' already exists"] run task successfully ------------------------------------------ test ddl create table ddl_reentrant.t4 (a int primary key, b int), is_reentrant: false restored_sql: CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT) table consistent_replicate_storage_file.check1 not exists for 22-th check, retry later check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c + set +x + tso='449348985879003137 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348985879003137 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449348985879003137 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:19:09 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1715117153.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:10 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002cceea0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348986208780291 MinTableBarrierTs:449348986470924291 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348986470924290, checkpointTs: 449348986470924290, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348986470924290, checkpointTs: 449348986470924290, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348986470924291, checkpointTs: 449348986470924291, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0f5b652-dfb6-4262-80a0-c4703121cf91 {"id":"e0f5b652-dfb6-4262-80a0-c4703121cf91","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2eed e0f5b652-dfb6-4262-80a0-c4703121cf91 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348986470924290,"min-table-barrier-ts":449348986470924291,"admin-job-type":0} /tidb/cdc/default/default/task/position/e0f5b652-dfb6-4262-80a0-c4703121cf91/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002cceea0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348986208780291 MinTableBarrierTs:449348986470924291 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348986470924290, checkpointTs: 449348986470924290, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348986470924290, checkpointTs: 449348986470924290, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348986470924291, checkpointTs: 449348986470924291, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0f5b652-dfb6-4262-80a0-c4703121cf91 {"id":"e0f5b652-dfb6-4262-80a0-c4703121cf91","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2eed e0f5b652-dfb6-4262-80a0-c4703121cf91 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348986470924290,"min-table-barrier-ts":449348986470924291,"admin-job-type":0} /tidb/cdc/default/default/task/position/e0f5b652-dfb6-4262-80a0-c4703121cf91/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002cceea0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348986208780291 MinTableBarrierTs:449348986470924291 AdminJobType:noop} span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348986470924290, checkpointTs: 449348986470924290, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348986470924290, checkpointTs: 449348986470924290, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348986470924291, checkpointTs: 449348986470924291, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e0f5b652-dfb6-4262-80a0-c4703121cf91 {"id":"e0f5b652-dfb6-4262-80a0-c4703121cf91","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2eed e0f5b652-dfb6-4262-80a0-c4703121cf91 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348986470924290,"min-table-barrier-ts":449348986470924291,"admin-job-type":0} /tidb/cdc/default/default/task/position/e0f5b652-dfb6-4262-80a0-c4703121cf91/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } check diff failed 2-th time, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) "is-owner": true, run task successfully table consistent_replicate_storage_file.check1 not exists for 23-th check, retry later changefeed is working normally rts: 449348986230538240->449348986230538240 checkpoint: 449348986151895100->449348986492682240 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/04/26 19:19:09.506 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT)"] [duration=83.626972ms] [namespace=default] [changefeed=94572c11-4be2-48f9-adbc-de25eb5fa07c] run task successfully ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) resume changefeed 94572c11-4be2-48f9-adbc-de25eb5fa07c from 449348986151895073 check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:12 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8f9afc8b-c35f-458d-833c-a504b0b21698 {"id":"8f9afc8b-c35f-458d-833c-a504b0b21698","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a212f60f7 8f9afc8b-c35f-458d-833c-a504b0b21698 /tidb/cdc/default/default/upstream/7362133736523670973 {"id":7362133736523670973,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8f9afc8b-c35f-458d-833c-a504b0b21698 {"id":"8f9afc8b-c35f-458d-833c-a504b0b21698","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a212f60f7 8f9afc8b-c35f-458d-833c-a504b0b21698 /tidb/cdc/default/default/upstream/7362133736523670973 {"id":7362133736523670973,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/8f9afc8b-c35f-458d-833c-a504b0b21698 {"id":"8f9afc8b-c35f-458d-833c-a504b0b21698","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a212f60f7 8f9afc8b-c35f-458d-833c-a504b0b21698 /tidb/cdc/default/default/upstream/7362133736523670973 {"id":7362133736523670973,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449348985879003137 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.17209.out cli changefeed create --start-ts=449348985879003137 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362133736523670973,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:19:12.668387701+08:00","start_ts":449348985879003137,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348985879003137,"checkpoint_ts":449348985879003137,"checkpoint_time":"2024-04-26 19:19:07.744"} PASS check diff successfully check diff failed 1-th time, retry later coverage: 2.5% of statements in github.com/pingcap/tiflow/... owner pid: 5740 owner id e0f5b652-dfb6-4262-80a0-c4703121cf91 process status: T ETCDCTL_API=3 etcdctl get /tidb/cdc/default/__cdc_meta__/owner --prefix | grep -v 'e0f5b652-dfb6-4262-80a0-c4703121cf91' /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2eed run task successfully process status: S nonempty select id, val from test.availability1 where id=2 and val=22 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file.check1 exists table consistent_replicate_storage_file.t2 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) run task failed 1-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 3609 0 --:--:-- --:--:-- --:--:-- 3622 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:19:07.744","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-04-26 19:19:14.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:19:07.744","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:19:14.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:19:07.744","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:19:14.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-04-26 19:19:07.744' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:19:07.744","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:19:14.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:19:07.744","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:19:14.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-04-26 19:19:14' + echo 'sink_checkpoint_ts is 2024-04-26' 19:19:07.744 sink_checkpoint_ts is 2024-04-26 19:19:07.744 ++ date -d '2024-04-26 19:19:07.744' +%s + checkpoint_timestamp=1714130347 ++ date -d '2024-04-26 19:19:14' +%s + current_timestamp=1714130354 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' check diff successfully + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later nonempty select id, val from test.availability1 where id=2 and val=22 run task successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886009c0008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:20918, start at 2024-04-26 19:19:14.224602762 +0800 CST m=+5.335535114 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:14.233 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:14.215 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:14.215 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... table consistent_replicate_storage_file.t2 not exists for 2-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68860a240017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:3896, start at 2024-04-26 19:19:14.851779006 +0800 CST m=+5.436039337 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:14.859 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:14.825 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:14.825 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... start tidb cluster in /tmp/tidb_cdc_test/autorandom Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:16 CST 2024] <<<<<< run test case capture_session_done_during_task success! >>>>>> check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit sql_check run sql_check, 127.0.0.1 run task successfully run test case test_stop_processor [Fri Apr 26 19:19:16 CST 2024] <<<<<< START cdc server in availability case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.availability.59295931.out server --log-file /tmp/tidb_cdc_test/availability/cdctest_stop_processor.log --log-level debug --data-dir /tmp/tidb_cdc_test/availability/cdc_datatest_stop_processor --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886009c0008 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:20918, start at 2024-04-26 19:19:14.224602762 +0800 CST m=+5.335535114 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:14.233 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:14.215 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:14.215 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688600900005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:20999, start at 2024-04-26 19:19:14.219193123 +0800 CST m=+5.281163070 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:14.227 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:14.212 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:14.212 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/simple/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/simple/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/simple/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/simple/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/simple/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table test.t1 exists + sleep 5 table consistent_replicate_storage_file.t2 not exists for 3-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68860a240017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:3896, start at 2024-04-26 19:19:14.851779006 +0800 CST m=+5.436039337 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:14.859 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:14.825 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:14.825 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68860a0c000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:3982, start at 2024-04-26 19:19:14.833932289 +0800 CST m=+5.349928920 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:14.841 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:14.819 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:14.819 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_finish/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_finish/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process 8826 exit for 1-th time... wait process 8826 exit for 2-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (8826) - No such process wait process 8826 exit for 3-th time... process 8826 already exit check_capture_count http://127.0.0.1:2379 0 parse error: Invalid numeric literal at line 1, column 6 run task successfully [Fri Apr 26 19:19:17 CST 2024] <<<<<< START cdc server in kill_owner_with_ddl case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.kill_owner_with_ddl.89728974.out server --log-file /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/kill_owner_with_ddl/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 changefeed is working normally rts: 449348987803402240->449348988327690240 checkpoint: 449348988065284100->449348988327690240 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql false [2024/04/26 19:19:12.180 +08:00] [INFO] [mysql_ddl_sink.go:119] ["Execute DDL failed, but error can be ignored"] [startTs=449348986151895068] [ddl="CREATE TABLE `ddl_reentrant`.`t4` (`a` INT PRIMARY KEY,`b` INT)"] [namespace=default] [changefeed=94572c11-4be2-48f9-adbc-de25eb5fa07c] [error="Error 1050 (42S01): Table 'ddl_reentrant.t4' already exists"] run task successfully ------------------------------------------ test ddl alter table ddl_reentrant.t3 exchange partition p0 with table ddl_reentrant.t4, is_reentrant: true restored_sql: ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4` check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.22227.out cli tso query --pd=http://127.0.0.1:2379 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688649e80006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:6760, start at 2024-04-26 19:19:18.916579076 +0800 CST m=+11.340578275 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:18.939 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:18.957 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:18.957 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688649e80006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:6760, start at 2024-04-26 19:19:18.916579076 +0800 CST m=+11.340578275 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:18.939 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:18.957 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:18.957 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886509c000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-xb6b7, pid:6783, start at 2024-04-26 19:19:19.36483747 +0800 CST m=+11.692471036 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:19.400 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:19.385 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:19.385 +0800 All versions after safe point can be accessed. (DO NOT EDIT) table consistent_replicate_storage_file.t2 not exists for 4-th check, retry later Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:19 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002dd83f0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348988830220291 MinTableBarrierTs:449348988830220292 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348988830220292, checkpointTs: 449348988830220292, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348988830220291, checkpointTs: 449348988830220291, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348988830220291, checkpointTs: 449348988830220291, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f16ed87-5f61-4bb4-957b-2cc5d5607c39 {"id":"0f16ed87-5f61-4bb4-957b-2cc5d5607c39","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2f38 0f16ed87-5f61-4bb4-957b-2cc5d5607c39 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348988830220291,"min-table-barrier-ts":449348988830220292,"admin-job-type":0} /tidb/cdc/default/default/task/position/0f16ed87-5f61-4bb4-957b-2cc5d5607c39/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002dd83f0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348988830220291 MinTableBarrierTs:449348988830220292 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348988830220292, checkpointTs: 449348988830220292, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348988830220291, checkpointTs: 449348988830220291, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348988830220291, checkpointTs: 449348988830220291, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f16ed87-5f61-4bb4-957b-2cc5d5607c39 {"id":"0f16ed87-5f61-4bb4-957b-2cc5d5607c39","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2f38 0f16ed87-5f61-4bb4-957b-2cc5d5607c39 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348988830220291,"min-table-barrier-ts":449348988830220292,"admin-job-type":0} /tidb/cdc/default/default/task/position/0f16ed87-5f61-4bb4-957b-2cc5d5607c39/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/0b0cc1de-0400-407b-bc84-46d5c1c5413a {UpstreamID:7362132776889089863 Namespace:default ID:0b0cc1de-0400-407b-bc84-46d5c1c5413a SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:15:33.644785793 +0800 CST StartTs:449348928366706692 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002dd83f0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348929743224842} {CheckpointTs:449348988830220291 MinTableBarrierTs:449348988830220292 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348988830220292, checkpointTs: 449348988830220292, state: Replicating span: {table_id:102,start_key:7480000000000000ff665f720000000000fa,end_key:7480000000000000ff665f730000000000fa}, resolvedTs: 449348988830220291, checkpointTs: 449348988830220291, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348988830220291, checkpointTs: 449348988830220291, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0f16ed87-5f61-4bb4-957b-2cc5d5607c39 {"id":"0f16ed87-5f61-4bb4-957b-2cc5d5607c39","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/meta/ticdc-delete-etcd-key-count 4 /tidb/cdc/default/__cdc_meta__/owner/22318f1a1dce2f38 0f16ed87-5f61-4bb4-957b-2cc5d5607c39 /tidb/cdc/default/default/changefeed/info/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"upstream-id":7362132776889089863,"namespace":"default","changefeed-id":"0b0cc1de-0400-407b-bc84-46d5c1c5413a","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:15:33.644785793+08:00","start-ts":449348928366706692,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348929743224842} /tidb/cdc/default/default/changefeed/status/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":449348988830220291,"min-table-barrier-ts":449348988830220292,"admin-job-type":0} /tidb/cdc/default/default/task/position/0f16ed87-5f61-4bb4-957b-2cc5d5607c39/0b0cc1de-0400-407b-bc84-46d5c1c5413a {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362132776889089863 {"id":7362132776889089863,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test cli capture list 2>&1 | grep '"is-owner": true' [Fri Apr 26 19:19:19 CST 2024] <<<<<< START cdc server in changefeed_finish case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_finish.52885290.out server --log-file /tmp/tidb_cdc_test/changefeed_finish/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_finish/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sink_hang/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sink_hang/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sink_hang/tiflash/log/proxy.log"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } changefeed is working normally rts: 449348988589834240->449348989113860100 checkpoint: 449348988589834240->449348989113860100 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/04/26 19:19:19.583 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4`"] [duration=159.94592ms] [namespace=default] [changefeed=94572c11-4be2-48f9-adbc-de25eb5fa07c] run task successfully + set +x + tso='449348988964700161 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348988964700161 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:19:21 CST 2024] <<<<<< START cdc server in simple case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.2227222274.out server --log-file /tmp/tidb_cdc_test/simple/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/simple/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:20 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0015bfdd0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976460955686 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66c0d102-a3d6-41fd-aa9c-906b0dafe413 {"id":"66c0d102-a3d6-41fd-aa9c-906b0dafe413","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830475 66c0d102-a3d6-41fd-aa9c-906b0dafe413 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976460955686,"admin-job-type":0} /tidb/cdc/default/default/task/position/66c0d102-a3d6-41fd-aa9c-906b0dafe413/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0015bfdd0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976460955686 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66c0d102-a3d6-41fd-aa9c-906b0dafe413 {"id":"66c0d102-a3d6-41fd-aa9c-906b0dafe413","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830475 66c0d102-a3d6-41fd-aa9c-906b0dafe413 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976460955686,"admin-job-type":0} /tidb/cdc/default/default/task/position/66c0d102-a3d6-41fd-aa9c-906b0dafe413/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {UpstreamID:7362133534577049106 Namespace:default ID:8d12ecb2-6a09-4d09-9b14-4d59459739c9 SinkURI:mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1 CreateTime:2024-04-26 19:18:24.635874324 +0800 CST StartTs:449348974547304450 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0015bfdd0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348974573518851} {CheckpointTs:449348976434741281 MinTableBarrierTs:449348976460955686 AdminJobType:noop} span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348976434741281, checkpointTs: 449348976434741281, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/66c0d102-a3d6-41fd-aa9c-906b0dafe413 {"id":"66c0d102-a3d6-41fd-aa9c-906b0dafe413","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20830475 66c0d102-a3d6-41fd-aa9c-906b0dafe413 /tidb/cdc/default/default/changefeed/info/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"upstream-id":7362133534577049106,"namespace":"default","changefeed-id":"8d12ecb2-6a09-4d09-9b14-4d59459739c9","sink-uri":"mysql://normal:123456@127.0.0.1:3306/?max-txn-row=1","create-time":"2024-04-26T19:18:24.635874324+08:00","start-ts":449348974547304450,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348974573518851} /tidb/cdc/default/default/changefeed/status/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":449348976434741281,"min-table-barrier-ts":449348976460955686,"admin-job-type":0} /tidb/cdc/default/default/task/position/66c0d102-a3d6-41fd-aa9c-906b0dafe413/8d12ecb2-6a09-4d09-9b14-4d59459739c9 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133534577049106 {"id":7362133534577049106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_capture_count http://127.0.0.1:2379 1 run task successfully table kill_owner_with_ddl.t1 exists table kill_owner_with_ddl.t2 exists table kill_owner_with_ddl.t3 not exists for 1-th check, retry later + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 resume changefeed 94572c11-4be2-48f9-adbc-de25eb5fa07c from 449348988629155852 check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c "is-owner": true, run task successfully table consistent_replicate_storage_file.t2 not exists for 5-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 3997 0 --:--:-- --:--:-- --:--:-- 4050 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:19:21.194","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:19:14.794","now_ts":"2024-04-26 19:19:21.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:19:21.194","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:19:14.794","now_ts":"2024-04-26' '19:19:21.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:19:21.194","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:19:14.794","now_ts":"2024-04-26' '19:19:21.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > owner pid: 5934 owner id 0f16ed87-5f61-4bb4-957b-2cc5d5607c39 [Fri Apr 26 19:19:22 CST 2024] <<<<<< START cdc server in sink_hang case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkExecDMLError=2*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sink_hang.80298031.out server --log-file /tmp/tidb_cdc_test/sink_hang/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sink_hang/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:22 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7b9be2e-de68-468b-8888-d77cc9376119 {"id":"d7b9be2e-de68-468b-8888-d77cc9376119","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a215789f1 d7b9be2e-de68-468b-8888-d77cc9376119 /tidb/cdc/default/default/upstream/7362133778593397932 {"id":7362133778593397932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7b9be2e-de68-468b-8888-d77cc9376119 {"id":"d7b9be2e-de68-468b-8888-d77cc9376119","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a215789f1 d7b9be2e-de68-468b-8888-d77cc9376119 /tidb/cdc/default/default/upstream/7362133778593397932 {"id":7362133778593397932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d7b9be2e-de68-468b-8888-d77cc9376119 {"id":"d7b9be2e-de68-468b-8888-d77cc9376119","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a215789f1 d7b9be2e-de68-468b-8888-d77cc9376119 /tidb/cdc/default/default/upstream/7362133778593397932 {"id":7362133778593397932,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table kill_owner_with_ddl.t3 exists check diff failed 1-th time, retry later run task failed 1-th time, retry later check diff failed 1-th time, retry later table consistent_replicate_storage_file.t2 not exists for 6-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) changefeed id: 0b0cc1de-0400-407b-bc84-46d5c1c5413a curl -s -d "cf-id=0b0cc1de-0400-407b-bc84-46d5c1c5413a&admin-job=1" http://127.0.0.1:8300/capture/owner/admin | grep true "status": true, run task successfully % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 89 100 35 100 54 1472 2271 --:--:-- --:--:-- --:--:-- 2347 { "status": true, "message": "" }nonempty select id, val from test.availability1 where id=4 and val=4 TEST FAILED: OUTPUT DOES NOT CONTAIN 'id:' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ run task failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:24 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/abc8772e-3080-41d1-a056-7ccf8feb71f5 {"id":"abc8772e-3080-41d1-a056-7ccf8feb71f5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2153670c abc8772e-3080-41d1-a056-7ccf8feb71f5 /tidb/cdc/default/default/upstream/7362133767193656002 {"id":7362133767193656002,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/abc8772e-3080-41d1-a056-7ccf8feb71f5 {"id":"abc8772e-3080-41d1-a056-7ccf8feb71f5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2153670c abc8772e-3080-41d1-a056-7ccf8feb71f5 /tidb/cdc/default/default/upstream/7362133767193656002 {"id":7362133767193656002,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/abc8772e-3080-41d1-a056-7ccf8feb71f5 {"id":"abc8772e-3080-41d1-a056-7ccf8feb71f5","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2153670c abc8772e-3080-41d1-a056-7ccf8feb71f5 /tidb/cdc/default/default/upstream/7362133767193656002 {"id":7362133767193656002,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.simple.cli.22318.out cli changefeed create --start-ts=449348988964700161 --sink-uri=mysql+ssl://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: 47f6342b-0346-46ef-9c87-34cf00861be1 Info: {"upstream_id":7362133767193656002,"namespace":"default","id":"47f6342b-0346-46ef-9c87-34cf00861be1","sink_uri":"mysql+ssl://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:19:24.641105274+08:00","start_ts":449348988964700161,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348988964700161,"checkpoint_ts":449348988964700161,"checkpoint_time":"2024-04-26 19:19:19.515"} PASS [2024/04/26 19:19:21.966 +08:00] [WARN] [diff.go:551] ["checksum is not equal"] [table=`test`.`itest`] [where="((TRUE) AND TRUE)"] ["source checksum"=4260364863] ["target checksum"=2001007844] ["get source checksum cost"=3.244059ms] ["get target checksum cost"=2.047029ms] coverage: 2.4% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:25 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5b702476-1ab9-4341-9b89-2c5b93087c32 {"id":"5b702476-1ab9-4341-9b89-2c5b93087c32","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a215008fb 5b702476-1ab9-4341-9b89-2c5b93087c32 /tidb/cdc/default/default/upstream/7362133771529221099 {"id":7362133771529221099,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5b702476-1ab9-4341-9b89-2c5b93087c32 {"id":"5b702476-1ab9-4341-9b89-2c5b93087c32","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a215008fb 5b702476-1ab9-4341-9b89-2c5b93087c32 /tidb/cdc/default/default/upstream/7362133771529221099 {"id":7362133771529221099,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/5b702476-1ab9-4341-9b89-2c5b93087c32 {"id":"5b702476-1ab9-4341-9b89-2c5b93087c32","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a215008fb 5b702476-1ab9-4341-9b89-2c5b93087c32 /tidb/cdc/default/default/upstream/7362133771529221099 {"id":7362133771529221099,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table consistent_replicate_storage_file.t2 exists check diff successfully wait process cdc.test exit for 1-th time... check_ts_forward 94572c11-4be2-48f9-adbc-de25eb5fa07c check diff successfully check diff failed 2-th time, retry later wait process cdc.test exit for 2-th time... + set +x succeed to verify meta placement rules TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 1-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/ddl_attributes/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:26 CST 2024] <<<<<< run test case kill_owner_with_ddl success! >>>>>> nonempty select id, val from test.availability1 where id=4 and val=4 run task successfully test_stop_processor pass wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... changefeed is working normally rts: 449348990424842240->449348990686724100 checkpoint: 449348990424842240->449348990686724100 run task successfully check_ddl_executed /tmp/tidb_cdc_test/ddl_reentrant/cdc.log /tmp/tidb_cdc_test/ddl_reentrant/ddl_temp.sql true [2024/04/26 19:19:23.374 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE `ddl_reentrant`.`t3` EXCHANGE PARTITION `p0` WITH TABLE `ddl_reentrant`.`t4`"] [duration=153.702556ms] [namespace=default] [changefeed=94572c11-4be2-48f9-adbc-de25eb5fa07c] run task successfully cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:27 CST 2024] <<<<<< run test case availability success! >>>>>> wait process cdc.test exit for 1-th time... [Fri Apr 26 19:19:27 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_file case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file.1119311195.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully check_changefeed_status 127.0.0.1:8300 31e2e4f7-7bb7-45cc-acdc-86a9c8eaeda2 normal last_error null + endpoint=127.0.0.1:8300 + changefeed_id=31e2e4f7-7bb7-45cc-acdc-86a9c8eaeda2 + expected_state=normal + field=last_error + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/31e2e4f7-7bb7-45cc-acdc-86a9c8eaeda2/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 727 0 --:--:-- --:--:-- --:--:-- 728 + info='{"state":"normal","resolved_ts":449348990575050759,"checkpoint_ts":449348990575050759}' + echo '{"state":"normal","resolved_ts":449348990575050759,"checkpoint_ts":449348990575050759}' {"state":"normal","resolved_ts":449348990575050759,"checkpoint_ts":449348990575050759} ++ echo '{"state":"normal","resolved_ts":449348990575050759,"checkpoint_ts":449348990575050759}' ++ jq -r .state VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886ccac0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:20984, start at 2024-04-26 19:19:27.282392276 +0800 CST m=+7.547070232 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:27.299 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:27.275 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:27.275 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_error ]] ++ echo '{"state":"normal","resolved_ts":449348990575050759,"checkpoint_ts":449348990575050759}' ++ jq -r .last_error.message + error_msg=null + [[ ! null =~ null ]] run task successfully check_changefeed_status 127.0.0.1:8300 31e2e4f7-7bb7-45cc-acdc-86a9c8eaeda2 normal last_warning null + endpoint=127.0.0.1:8300 + changefeed_id=31e2e4f7-7bb7-45cc-acdc-86a9c8eaeda2 + expected_state=normal + field=last_warning + error_pattern=null ++ curl 127.0.0.1:8300/api/v2/changefeeds/31e2e4f7-7bb7-45cc-acdc-86a9c8eaeda2/status % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 86 100 86 0 0 690 0 --:--:-- --:--:-- --:--:-- 688 100 86 100 86 0 0 689 0 --:--:-- --:--:-- --:--:-- 688 + info='{"state":"normal","resolved_ts":449348990692753418,"checkpoint_ts":449348990692753418}' + echo '{"state":"normal","resolved_ts":449348990692753418,"checkpoint_ts":449348990692753418}' {"state":"normal","resolved_ts":449348990692753418,"checkpoint_ts":449348990692753418} ++ echo '{"state":"normal","resolved_ts":449348990692753418,"checkpoint_ts":449348990692753418}' ++ jq -r .state wait process cdc.test exit for 2-th time... + state=normal + [[ ! normal == \n\o\r\m\a\l ]] + [[ -z last_warning ]] ++ echo '{"state":"normal","resolved_ts":449348990692753418,"checkpoint_ts":449348990692753418}' ++ jq -r .last_warning.message + error_msg=null + [[ ! null =~ null ]] run task successfully table sink_hang.t1 not exists for 1-th check, retry later TEST FAILED: OUTPUT DOES NOT CONTAIN 'id: 1' ____________________________________ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ check data failed 2-th time, retry later start tidb cluster in /tmp/tidb_cdc_test/ddl_attributes Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:28 CST 2024] <<<<<< run test case ddl_reentrant success! >>>>>> VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886ccac0017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:20984, start at 2024-04-26 19:19:27.282392276 +0800 CST m=+7.547070232 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:27.299 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:27.275 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:27.275 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6886ce3c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:21020, start at 2024-04-26 19:19:27.390041862 +0800 CST m=+7.544157974 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:27.409 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:27.424 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:27.424 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... table sink_hang.t1 exists table sink_hang.t2 exists check data successfully Logging trace to /tmp/tidb_cdc_test/autorandom/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/autorandom/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/autorandom/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } check diff failed 1-th time, retry later pass check, checkpoint tso not forward after 10s run task successfully wait process 7165 exit for 1-th time... wait process 7165 exit for 2-th time... wait process 7165 exit for 3-th time... wait process 7165 exit for 4-th time... wait process 7165 exit for 5-th time... /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/_utils/kill_cdc_pid: line 19: kill: (7165) - No such process wait process 7165 exit for 6-th time... process 7165 already exit [Fri Apr 26 19:19:26 CST 2024] <<<<<< START cdc server in ddl_only_block_related_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_only_block_related_table.75417543.out server --log-file /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_only_block_related_table/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133426535730208 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:04.112622636 +0800 CST StartTs:449348969158148100 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037ee870 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348969197207553} {CheckpointTs:449348973352189953 MinTableBarrierTs:449348991649841154 AdminJobType:noop} span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9f527541-7104-42ec-a2be-bc1db0046a2a {"id":"9f527541-7104-42ec-a2be-bc1db0046a2a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20185387 9f527541-7104-42ec-a2be-bc1db0046a2a /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133426535730208,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:18:04.112622636+08:00","start-ts":449348969158148100,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348969197207553} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348973352189953,"min-table-barrier-ts":449348991649841154,"admin-job-type":0} /tidb/cdc/default/default/task/position/9f527541-7104-42ec-a2be-bc1db0046a2a/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133426535730208 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:04.112622636 +0800 CST StartTs:449348969158148100 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037ee870 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348969197207553} {CheckpointTs:449348973352189953 MinTableBarrierTs:449348991649841154 AdminJobType:noop} span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9f527541-7104-42ec-a2be-bc1db0046a2a {"id":"9f527541-7104-42ec-a2be-bc1db0046a2a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20185387 9f527541-7104-42ec-a2be-bc1db0046a2a /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133426535730208,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:18:04.112622636+08:00","start-ts":449348969158148100,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348969197207553} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348973352189953,"min-table-barrier-ts":449348991649841154,"admin-job-type":0} /tidb/cdc/default/default/task/position/9f527541-7104-42ec-a2be-bc1db0046a2a/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/ddl-only-block-related-table {UpstreamID:7362133426535730208 Namespace:default ID:ddl-only-block-related-table SinkURI:mysql://root@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:04.112622636 +0800 CST StartTs:449348969158148100 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0037ee870 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348969197207553} {CheckpointTs:449348973352189953 MinTableBarrierTs:449348991649841154 AdminJobType:noop} span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348973352189953, checkpointTs: 449348973352189953, state: Preparing *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9f527541-7104-42ec-a2be-bc1db0046a2a {"id":"9f527541-7104-42ec-a2be-bc1db0046a2a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20185387 9f527541-7104-42ec-a2be-bc1db0046a2a /tidb/cdc/default/default/changefeed/info/ddl-only-block-related-table {"upstream-id":7362133426535730208,"namespace":"default","changefeed-id":"ddl-only-block-related-table","sink-uri":"mysql://root@127.0.0.1:3306/","create-time":"2024-04-26T19:18:04.112622636+08:00","start-ts":449348969158148100,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"none","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348969197207553} /tidb/cdc/default/default/changefeed/status/ddl-only-block-related-table {"checkpoint-ts":449348973352189953,"min-table-barrier-ts":449348991649841154,"admin-job-type":0} /tidb/cdc/default/default/task/position/9f527541-7104-42ec-a2be-bc1db0046a2a/ddl-only-block-related-table {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133426535730208 {"id":7362133426535730208,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check diff successfully check_ts_forward ddl-only-block-related-table wait process cdc.test exit for 1-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:30 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/80ff8a4c-be9f-4961-b043-10756da8a0ca {UpstreamID:7362133522636522512 Namespace:default ID:80ff8a4c-be9f-4961-b043-10756da8a0ca SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:21.336132747 +0800 CST StartTs:449348973688520711 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003632f30 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348973701627910} {CheckpointTs:449348991697813513 MinTableBarrierTs:449348991697813513 AdminJobType:noop} span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fee71f88-419c-4aaa-bea8-3334015adb80 {"id":"fee71f88-419c-4aaa-bea8-3334015adb80","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20705b2b fee71f88-419c-4aaa-bea8-3334015adb80 /tidb/cdc/default/default/changefeed/info/80ff8a4c-be9f-4961-b043-10756da8a0ca {"upstream-id":7362133522636522512,"namespace":"default","changefeed-id":"80ff8a4c-be9f-4961-b043-10756da8a0ca","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:18:21.336132747+08:00","start-ts":449348973688520711,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348973701627910} /tidb/cdc/default/default/changefeed/status/80ff8a4c-be9f-4961-b043-10756da8a0ca {"checkpoint-ts":449348991697813513,"min-table-barrier-ts":449348991697813513,"admin-job-type":0} /tidb/cdc/default/default/task/position/fee71f88-419c-4aaa-bea8-3334015adb80/80ff8a4c-be9f-4961-b043-10756da8a0ca {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133522636522512 {"id":7362133522636522512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/80ff8a4c-be9f-4961-b043-10756da8a0ca {UpstreamID:7362133522636522512 Namespace:default ID:80ff8a4c-be9f-4961-b043-10756da8a0ca SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:21.336132747 +0800 CST StartTs:449348973688520711 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003632f30 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348973701627910} {CheckpointTs:449348991697813513 MinTableBarrierTs:449348991697813513 AdminJobType:noop} span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fee71f88-419c-4aaa-bea8-3334015adb80 {"id":"fee71f88-419c-4aaa-bea8-3334015adb80","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20705b2b fee71f88-419c-4aaa-bea8-3334015adb80 /tidb/cdc/default/default/changefeed/info/80ff8a4c-be9f-4961-b043-10756da8a0ca {"upstream-id":7362133522636522512,"namespace":"default","changefeed-id":"80ff8a4c-be9f-4961-b043-10756da8a0ca","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:18:21.336132747+08:00","start-ts":449348973688520711,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region+ grep -q 'failed to get info:' -per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348973701627910} /tidb/cdc/default/default/changefeed/status/80ff8a4c-be9f-4961-b043-10756da8a0ca {"checkpoint-ts":449348991697813513,"min-table-barrier-ts":449348991697813513,"admin-job-type":0} /tidb/cdc/default/default/task/position/fee71f88-419c-4aaa-bea8-3334015adb80/80ff8a4c-be9f-4961-b043-10756da8a0ca {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133522636522512 {"id":7362133522636522512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/80ff8a4c-be9f-4961-b043-10756da8a0ca {UpstreamID:7362133522636522512 Namespace:default ID:80ff8a4c-be9f-4961-b043-10756da8a0ca SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:18:21.336132747 +0800 CST StartTs:449348973688520711 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc003632f30 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449348973701627910} {CheckpointTs:449348991697813513 MinTableBarrierTs:449348991697813513 AdminJobType:noop} span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449348991697813514, checkpointTs: 449348991697813513, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449348991697813513, checkpointTs: 449348991697813513, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fee71f88-419c-4aaa-bea8-3334015adb80 {"id":"fee71f88-419c-4aaa-bea8-3334015adb80","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a20705b2b fee71f88-419c-4aaa-bea8-3334015adb80 /tidb/cdc/default/default/changefeed/info/80ff8a4c-be9f-4961-b043-10756da8a0ca {"upstream-id":7362133522636522512,"namespace":"default","changefeed-id":"80ff8a4c-be9f-4961-b043-10756da8a0ca","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:18:21.336132747+08:00","start-ts":449348973688520711,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo","use-file-backend":false,"compression":"lz4","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449348973701627910} /tidb/cdc/default/default/changefeed/status/80ff8a4c-be9f-4961-b043-10756da8a0ca {"checkpoint-ts":449348991697813513,"min-table-barrier-ts":449348991697813513,"admin-job-type":0} /tidb/cdc/default/default/task/position/fee71f88-419c-4aaa-bea8-3334015adb80/80ff8a4c-be9f-4961-b043-10756da8a0ca {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362133522636522512 {"id":7362133522636522512,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:19:31 CST 2024] <<<<<< run test case simple success! >>>>>> [Fri Apr 26 19:19:32 CST 2024] <<<<<< START cdc server in autorandom case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.autorandom.2228122283.out server --log-file /tmp/tidb_cdc_test/autorandom/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/autorandom/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 changefeed is working normally rts: 449348973352189950->449348992174129150 checkpoint: 449348973352189950->449348992174129150 run task successfully wait process cdc.test exit for 1-th time... check diff failed 2-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:19:34 CST 2024] <<<<<< run test case ddl_only_block_related_table success! >>>>>> check diff failed 3-th time, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:35 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d711169-49a6-4479-85c0-aea23aeacb36 {"id":"2d711169-49a6-4479-85c0-aea23aeacb36","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21856bf5 2d711169-49a6-4479-85c0-aea23aeacb36 /tidb/cdc/default/default/upstream/7362133822283458954 {"id":7362133822283458954,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d711169-49a6-4479-85c0-aea23aeacb36 {"id":"2d711169-49a6-4479-85c0-aea23aeacb36","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21856bf5 2d711169-49a6-4479-85c0-aea23aeacb36 /tidb/cdc/default/default/upstream/7362133822283458954 {"id":7362133822283458954,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2d711169-49a6-4479-85c0-aea23aeacb36 {"id":"2d711169-49a6-4479-85c0-aea23aeacb36","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21856bf5 2d711169-49a6-4479-85c0-aea23aeacb36 /tidb/cdc/default/default/upstream/7362133822283458954 {"id":7362133822283458954,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: 2db25baf-1690-49f6-a3ef-72ce96d6c87f Info: {"upstream_id":7362133822283458954,"namespace":"default","id":"2db25baf-1690-49f6-a3ef-72ce96d6c87f","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:19:36.052967634+08:00","start_ts":449348993253376004,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348993253376004,"checkpoint_ts":449348993253376004,"checkpoint_time":"2024-04-26 19:19:35.875"} table autorandom_test.table_a not exists for 1-th check, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/owner_remove_table_error/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/http_proxies + CDC_BINARY=cdc.test + SINK_TYPE=mysql + TEST_HOST_LIST=(UP_TIDB_HOST UP_PD_HOST_{1..3} UP_TIKV_HOST_{1..3}) + lan_addrs=($(hostname -I)) ++ hostname -I + lan_addr=10.233.127.54 + export UP_TIDB_HOST=10.233.127.54 UP_PD_HOST_1=10.233.127.54 UP_PD_HOST_2=10.233.127.54 UP_PD_HOST_3=10.233.127.54 UP_TIKV_HOST_1=10.233.127.54 UP_TIKV_HOST_2=10.233.127.54 UP_TIKV_HOST_3=10.233.127.54 + UP_TIDB_HOST=10.233.127.54 + UP_PD_HOST_1=10.233.127.54 + UP_PD_HOST_2=10.233.127.54 + UP_PD_HOST_3=10.233.127.54 + UP_TIKV_HOST_1=10.233.127.54 + UP_TIKV_HOST_2=10.233.127.54 + UP_TIKV_HOST_3=10.233.127.54 + proxy_pid= ++ shuf -i 10081-20081 -n1 + proxy_port=11949 + trap 'stop_tidb_cluster && stop_proxy' EXIT + '[' mysql == mysql ']' + prepare + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_TIDB_HOST + echo 'UP_TIDB_HOST 10.233.127.54' UP_TIDB_HOST 10.233.127.54 + case $(printenv $host) in ++ printenv UP_TIDB_HOST + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_PD_HOST_1 + echo 'UP_PD_HOST_1 10.233.127.54' UP_PD_HOST_1 10.233.127.54 + case $(printenv $host) in ++ printenv UP_PD_HOST_1 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_PD_HOST_2 + echo 'UP_PD_HOST_2 10.233.127.54' UP_PD_HOST_2 10.233.127.54 + case $(printenv $host) in ++ printenv UP_PD_HOST_2 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_PD_HOST_3 + echo 'UP_PD_HOST_3 10.233.127.54' UP_PD_HOST_3 10.233.127.54 + case $(printenv $host) in ++ printenv UP_PD_HOST_3 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_TIKV_HOST_1 + echo 'UP_TIKV_HOST_1 10.233.127.54' UP_TIKV_HOST_1 10.233.127.54 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_1 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_TIKV_HOST_2 + echo 'UP_TIKV_HOST_2 10.233.127.54' UP_TIKV_HOST_2 10.233.127.54 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_2 + for host in '"${TEST_HOST_LIST[@]}"' ++ printenv UP_TIKV_HOST_3 + echo 'UP_TIKV_HOST_3 10.233.127.54' UP_TIKV_HOST_3 10.233.127.54 + case $(printenv $host) in ++ printenv UP_TIKV_HOST_3 + rm -rf /tmp/tidb_cdc_test/http_proxies + mkdir -p /tmp/tidb_cdc_test/http_proxies + stop_tidb_cluster check diff failed 4-th time, retry later =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_cdc_cluster/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table autorandom_test.table_a not exists for 2-th check, retry later start tidb cluster in /tmp/tidb_cdc_test/owner_remove_table_error Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68877d600012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:23043, start at 2024-04-26 19:19:38.610697548 +0800 CST m=+5.265123840 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:38.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:38.584 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:38.584 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) check diff failed 5-th time, retry later + start_tidb_cluster --workdir /tmp/tidb_cdc_test/http_proxies The 1 times to try to start tidb cluster... Verifying downstream PD is started... table autorandom_test.table_a exists check diff successfully wait process cdc.test exit for 1-th time... start tidb cluster in /tmp/tidb_cdc_test/multi_cdc_cluster Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68877d600012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:23043, start at 2024-04-26 19:19:38.610697548 +0800 CST m=+5.265123840 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:38.617 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:38.584 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:38.584 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68877fa00002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-jtl7b, pid:23107, start at 2024-04-26 19:19:38.729692077 +0800 CST m=+5.309905272 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:38.737 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:38.728 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:38.728 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/ddl_attributes/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/log/proxy.log"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/ddl_attributes/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/cdc_server_tips/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... wait process cdc.test exit for 3-th time... check diff failed 6-th time, retry later Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:19:42 CST 2024] <<<<<< run test case autorandom success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/http_proxies Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.24352.out cli tso query --pd=http://127.0.0.1:2379 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/cdc_server_tips Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check diff failed 7-th time, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x + tso='449348995378839553 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348995378839553 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:19:45 CST 2024] <<<<<< START cdc server in ddl_attributes case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.2438924391.out server --log-file /tmp/tidb_cdc_test/ddl_attributes/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/ddl_attributes/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.127.54' (111) check diff failed 8-th time, retry later Verifying downstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.127.54' (111) check diff failed 9-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:49 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b {"id":"4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21b8d6f4 4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b /tidb/cdc/default/default/upstream/7362133882567028842 {"id":7362133882567028842,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b {"id":"4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21b8d6f4 4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b /tidb/cdc/default/default/upstream/7362133882567028842 {"id":7362133882567028842,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b {"id":"4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21b8d6f4 4dbd9e5f-5a0f-442b-bb1b-3dad227f2a0b /tidb/cdc/default/default/upstream/7362133882567028842 {"id":7362133882567028842,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.ddl_attributes.cli.24436.out cli changefeed create --start-ts=449348995378839553 --sink-uri=mysql://root@127.0.0.1:3306/ Create changefeed successfully! ID: 87f9dae0-5ee2-455e-9830-f4b874db655c Info: {"upstream_id":7362133882567028842,"namespace":"default","id":"87f9dae0-5ee2-455e-9830-f4b874db655c","sink_uri":"mysql://root@127.0.0.1:3306/","create_time":"2024-04-26T19:19:50.14475638+08:00","start_ts":449348995378839553,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348995378839553,"checkpoint_ts":449348995378839553,"checkpoint_time":"2024-04-26 19:19:43.983"} PASS ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) coverage: 2.4% of statements in github.com/pingcap/tiflow/... Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688825700015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:10092, start at 2024-04-26 19:19:49.362179281 +0800 CST m=+5.101398527 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:49.369 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:49.340 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:49.340 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688825700015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:10092, start at 2024-04-26 19:19:49.362179281 +0800 CST m=+5.101398527 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:49.369 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:49.340 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:49.340 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688826100015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-kfqzr, pid:10180, start at 2024-04-26 19:19:49.416883712 +0800 CST m=+5.076986634 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:49.423 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:49.380 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:49.380 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/owner_remove_table_error/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.127.54' (111) check diff failed 10-th time, retry later + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/generate_column/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '10.233.127.54' (111) =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/changefeed_auto_stop/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Fri Apr 26 19:19:52 CST 2024] <<<<<< START cdc server in owner_remove_table_error case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/OwnerRemoveTableError=1*return(true)' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.owner_remove_table_error.1142611428.out server --log-file /tmp/tidb_cdc_test/owner_remove_table_error/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/owner_remove_table_error/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68883e700013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:10825, start at 2024-04-26 19:19:50.983296206 +0800 CST m=+5.246982694 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:50.990 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:50.991 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:50.991 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68883e700013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:10825, start at 2024-04-26 19:19:50.983296206 +0800 CST m=+5.246982694 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:50.990 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:50.991 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:50.991 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688840c8000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-snw4g, pid:10902, start at 2024-04-26 19:19:51.107454287 +0800 CST m=+5.321535750 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:51.115 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:51.090 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:51.090 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... wait process cdc.test exit for 1-th time... Logging trace to /tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/multi_cdc_cluster/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } start tidb cluster in /tmp/tidb_cdc_test/changefeed_auto_stop Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... wait process cdc.test exit for 2-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68885eb0000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:7116, start at 2024-04-26 19:19:53.016623939 +0800 CST m=+6.270226073 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:53.025 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:53.004 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:53.004 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68885eb0000a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:7116, start at 2024-04-26 19:19:53.016623939 +0800 CST m=+6.270226073 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:53.025 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:53.004 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:53.004 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68885f740013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:7208, start at 2024-04-26 19:19:53.078663391 +0800 CST m=+6.246155885 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:53.086 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:53.053 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:53.053 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/http_proxies/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/http_proxies/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["10.233.127.54:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash-proxy.toml"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/http_proxies/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } wait process cdc.test exit for 3-th time... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:19:55 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d3c98dcc-87b9-4610-a9e7-9ca86ac1a491 {"id":"d3c98dcc-87b9-4610-a9e7-9ca86ac1a491","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21e17ef3 d3c98dcc-87b9-4610-a9e7-9ca86ac1a491 /tidb/cdc/default/default/upstream/7362133921286105317 {"id":7362133921286105317,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d3c98dcc-87b9-4610-a9e7-9ca86ac1a491 {"id":"d3c98dcc-87b9-4610-a9e7-9ca86ac1a491","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21e17ef3 d3c98dcc-87b9-4610-a9e7-9ca86ac1a491 /tidb/cdc/default/default/upstream/7362133921286105317 {"id":7362133921286105317,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d3c98dcc-87b9-4610-a9e7-9ca86ac1a491 {"id":"d3c98dcc-87b9-4610-a9e7-9ca86ac1a491","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21e17ef3 d3c98dcc-87b9-4610-a9e7-9ca86ac1a491 /tidb/cdc/default/default/upstream/7362133921286105317 {"id":7362133921286105317,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:19:55 CST 2024] <<<<<< run test case sink_hang success! >>>>>> + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12141.out cli tso query --pd=http://127.0.0.1:2379 VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68888b7c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:23430, start at 2024-04-26 19:19:55.881232977 +0800 CST m=+5.365133870 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:55.890 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:55.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:55.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/generate_column Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... + start_proxy + echo 'dumpling grpc packet to /tmp/tidb_cdc_test/http_proxies/test_proxy.log...' dumpling grpc packet to /tmp/tidb_cdc_test/http_proxies/test_proxy.log... + proxy_pid=8451 + echo 'proxy port: 11949' proxy port: 11949 + echo 'proxy pid: 8451' proxy pid: 8451 + sleep 5 + GO111MODULE=on + WORK_DIR=/tmp/tidb_cdc_test/http_proxies + go run /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/http_proxies/run-proxy.go --port=11949 go: downloading google.golang.org/grpc v1.62.1 go: downloading go.uber.org/zap v1.27.0 go: downloading github.com/bradleyjkemp/grpc-tools v0.2.5 go: downloading github.com/pingcap/log v1.1.1-0.20240314023424-862ccc32f18d go: downloading gopkg.in/natefinch/lumberjack.v2 v2.2.1 go: downloading golang.org/x/net v0.24.0 go: downloading github.com/improbable-eng/grpc-web v0.12.0 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/pkg/errors v0.9.1 go: downloading golang.org/x/sys v0.19.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f go: downloading github.com/gorilla/websocket v1.5.1 go: downloading github.com/rs/cors v1.7.0 go: downloading golang.org/x/text v0.14.0 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 go: downloading google.golang.org/protobuf v1.33.0 go: downloading github.com/golang/protobuf v1.5.4 go: downloading google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 check_redo_resolved_ts 80ff8a4c-be9f-4961-b043-10756da8a0ca 449348997871304705 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/80ff8a4c-be9f-4961-b043-10756da8a0ca/meta global resolved ts 449348995394830343 not forward to 449348997871304705 run task failed 1-th time, retry later check_redo_resolved_ts 80ff8a4c-be9f-4961-b043-10756da8a0ca 449348997871304705 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file/cdc_data/redo/80ff8a4c-be9f-4961-b043-10756da8a0ca/meta run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + set +x + tso='449348998618415105 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348998618415105 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2024/04/26 19:19:57.918 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/04/26 19:19:57.919 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/04/26 19:19:57.919 +08:00] [INFO] [redo.go:163] ["apply redo log starts"] [checkpointTs=449348992104136708] [resolvedTs=449348998015483908] [2024/04/26 19:19:57.921 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.852005ms] [2024/04/26 19:19:57.923 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:19:57.923 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:19:57.931 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CNO_ENGINE_SUBSTITUTION%2CIGNORE_SPACE%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:19:57.934 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/04/26 19:19:57.935 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/04/26 19:19:57.936 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:19:57.936 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:19:57.941 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:19:57.944 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] table ddl_attributes.attributes_t1_new not exists for 1-th check, retry later [2024/04/26 19:19:58.489 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=569.592758ms] [2024/04/26 19:19:58.489 +08:00] [ERROR] [redo.go:268] ["ignore unsupported DDL"] [ddl="{\"StartTs\":449348992091029516,\"CommitTs\":449348992104136708,\"Query\":\"ALTER TABLE `consistent_replicate_storage_file`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_file`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_storage_file\",\"tbl-name\":\"t1\",\"tbl-id\":108,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68888b7c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:23430, start at 2024-04-26 19:19:55.881232977 +0800 CST m=+5.365133870 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:55.890 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:55.871 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:55.871 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68888b78000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:23507, start at 2024-04-26 19:19:55.882720567 +0800 CST m=+5.294802767 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:21:55.892 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:19:55.870 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:09:55.870 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/cdc_server_tips/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table owner_remove_table_error.finished_mark not exists for 1-th check, retry later table ddl_attributes.attributes_t1_new not exists for 2-th check, retry later [Fri Apr 26 19:19:58 CST 2024] <<<<<< START cdc server in multi_cdc_cluster case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.1219212194.out server --log-file /tmp/tidb_cdc_test/multi_cdc_cluster/cdcmult_cdc.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_cdc_cluster/cdc_datamult_cdc.server1 --cluster-id test1 --addr 127.0.0.1:8300 + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:19:59.494 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=449348998015483908] [2024/04/26 19:19:59.495 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=449348998015483908] Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:20:00.197 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:118,start_key:7480000000000000ff765f720000000000fa,end_key:7480000000000000ff765f730000000000fa}] [checkpointTs=449348998015483908] [2024/04/26 19:20:00.197 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:118,start_key:7480000000000000ff765f720000000000fa,end_key:7480000000000000ff765f730000000000fa}] [checkpointTs=449348998015483908] [2024/04/26 19:20:00.298 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=449348998015483908] [2024/04/26 19:20:00.298 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=449348998015483908] [2024/04/26 19:20:00.398 +08:00] [INFO] [redo.go:219] ["apply redo log finishes"] [appliedLogCount=5008] [appliedDDLCount=0] [currentCheckpoint=449348998015483908] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/04/26 19:20:00.398 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] Apply redo log successfully table owner_remove_table_error.finished_mark not exists for 2-th check, retry later check diff successfully [Fri Apr 26 19:20:00 CST 2024] <<<<<< run test case consistent_replicate_storage_file success! >>>>>> count(*) 5000 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.cli.24735.out cli tso query --pd=http://127.0.0.1:2379 ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.attributes_t1_new exists table ddl_attributes.finish_mark not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:02 GMT < Content-Length: 605 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/4cba1a0e-8a14-4e86-928b-62cf3eb7bbca {"id":"4cba1a0e-8a14-4e86-928b-62cf3eb7bbca","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/22318f1a21e94d0c 4cba1a0e-8a14-4e86-928b-62cf3eb7bbca /tidb/cdc/test1/default/upstream/7362133933378672677 {"id":7362133933378672677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/4cba1a0e-8a14-4e86-928b-62cf3eb7bbca {"id":"4cba1a0e-8a14-4e86-928b-62cf3eb7bbca","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/22318f1a21e94d0c 4cba1a0e-8a14-4e86-928b-62cf3eb7bbca /tidb/cdc/test1/default/upstream/7362133933378672677 {"id":7362133933378672677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test1/__cdc_meta__/capture/4cba1a0e-8a14-4e86-928b-62cf3eb7bbca {"id":"4cba1a0e-8a14-4e86-928b-62cf3eb7bbca","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/test1/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test1/__cdc_meta__/owner/22318f1a21e94d0c 4cba1a0e-8a14-4e86-928b-62cf3eb7bbca /tidb/cdc/test1/default/upstream/7362133933378672677 {"id":7362133933378672677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:20:02 CST 2024] <<<<<< START cdc server in multi_cdc_cluster case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.1223912241.out server --log-file /tmp/tidb_cdc_test/multi_cdc_cluster/cdcmult_cdc.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/multi_cdc_cluster/cdc_datamult_cdc.server2 --cluster-id test2 --addr 127.0.0.1:8301 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + set +x + tso='449348999871725569 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449348999871725569 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x try a VALID cdc server command [Fri Apr 26 19:20:02 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.2476724769.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table owner_remove_table_error.finished_mark exists check diff successfully + export http_proxy=http://127.0.0.1:11949 + http_proxy=http://127.0.0.1:11949 + export https_proxy=http://127.0.0.1:11949 + https_proxy=http://127.0.0.1:11949 + echo 'try to connect pd cluster via proxy, pd addr: 10.233.127.54:2379' try to connect pd cluster via proxy, pd addr: 10.233.127.54:2379 + ensure 10 curl http://10.233.127.54:2379/ curl http://10.233.127.54:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 127.0.0.1:11949; Connection refused run task failed 1-th time, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) wait process cdc.test exit for 1-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table ddl_attributes.finish_mark not exists for 2-th check, retry later wait process cdc.test exit for 2-th time... curl http://10.233.127.54:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 127.0.0.1:11949; Connection refused run task failed 2-th time, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:20:04 CST 2024] <<<<<< run test case owner_remove_table_error success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:05 GMT < Content-Length: 605 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/d7afed50-df75-4aab-abc3-18dd9d976c50 {"id":"d7afed50-df75-4aab-abc3-18dd9d976c50","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/22318f1a21e94d27 d7afed50-df75-4aab-abc3-18dd9d976c50 /tidb/cdc/test2/default/upstream/7362133933378672677 {"id":7362133933378672677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/d7afed50-df75-4aab-abc3-18dd9d976c50 {"id":"d7afed50-df75-4aab-abc3-18dd9d976c50","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/22318f1a21e94d27 d7afed50-df75-4aab-abc3-18dd9d976c50 /tidb/cdc/test2/default/upstream/7362133933378672677 {"id":7362133933378672677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/test2/__cdc_meta__/capture/d7afed50-df75-4aab-abc3-18dd9d976c50 {"id":"d7afed50-df75-4aab-abc3-18dd9d976c50","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/test2/__cdc_meta__/meta/meta-version 1 /tidb/cdc/test2/__cdc_meta__/owner/22318f1a21e94d27 d7afed50-df75-4aab-abc3-18dd9d976c50 /tidb/cdc/test2/default/upstream/7362133933378672677 {"id":7362133933378672677,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12294.out cli changefeed create --start-ts=449348998618415105 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server http://127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_cdc_cluster/conf/changefeed1.toml table ddl_attributes.finish_mark not exists for 3-th check, retry later Create changefeed successfully! ID: 01ff8bde-3ac9-4da9-b390-4a777ce86849 Info: {"upstream_id":7362133933378672677,"namespace":"default","id":"01ff8bde-3ac9-4da9-b390-4a777ce86849","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:20:05.525356642+08:00","start_ts":449348998618415105,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.multi_cdc1"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348998618415105,"checkpoint_ts":449348998618415105,"checkpoint_time":"2024-04-26 19:19:56.341"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:05 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/223d9b52-3969-4d17-9f3f-71427d647be1 {"id":"223d9b52-3969-4d17-9f3f-71427d647be1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21f5ecff 223d9b52-3969-4d17-9f3f-71427d647be1 /tidb/cdc/default/default/upstream/7362133949196932980 {"id":7362133949196932980,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/223d9b52-3969-4d17-9f3f-71427d647be1 {"id":"223d9b52-3969-4d17-9f3f-71427d647be1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21f5ecff 223d9b52-3969-4d17-9f3f-71427d647be1 /tidb/cdc/default/default/upstream/7362133949196932980 {"id":7362133949196932980,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/223d9b52-3969-4d17-9f3f-71427d647be1 {"id":"223d9b52-3969-4d17-9f3f-71427d647be1","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a21f5ecff 223d9b52-3969-4d17-9f3f-71427d647be1 /tidb/cdc/default/default/upstream/7362133949196932980 {"id":7362133949196932980,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.multi_cdc_cluster.cli.12320.out cli changefeed create --start-ts=449348998618415105 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server http://127.0.0.1:8301 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_cdc_cluster/conf/changefeed2.toml Create changefeed successfully! ID: b6cffd4b-a154-4859-a35b-a17669e56867 Info: {"upstream_id":7362133933378672677,"namespace":"default","id":"b6cffd4b-a154-4859-a35b-a17669e56867","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:20:07.305555518+08:00","start_ts":449348998618415105,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["test.multi_cdc2"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449348998618415105,"checkpoint_ts":449348998618415105,"checkpoint_time":"2024-04-26 19:19:56.341"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688936980002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:8546, start at 2024-04-26 19:20:06.831695278 +0800 CST m=+7.373015950 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:06.854 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:06.822 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:06.822 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688936980002 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:8546, start at 2024-04-26 19:20:06.831695278 +0800 CST m=+7.373015950 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:06.854 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:06.822 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:06.822 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688937b40006 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-rm9wh, pid:8589, start at 2024-04-26 19:20:06.90662293 +0800 CST m=+7.358358681 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:06.934 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:06.944 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:06.944 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/error.log arg matches is ArgMatches { args: {"addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/changefeed_auto_stop/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table ddl_attributes.finish_mark not exists for 4-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } curl http://10.233.127.54:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to 127.0.0.1:11949; Connection refused run task failed 3-th time, retry later [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container [Pipeline] } + set +x table test.multi_cdc1 exists table test.multi_cdc2 exists [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } check diff failed 1-th time, retry later [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table ddl_attributes.finish_mark not exists for 5-th check, retry later time="2024-04-26T19:20:10+08:00" level=info msg="Listening on 127.0.0.1:11949" time="2024-04-26T19:20:10+08:00" level=info msg="Not intercepting TLS connections" ***************** properties ***************** "readproportion"="0" "mysql.port"="4000" "operationcount"="0" "insertproportion"="0" "workload"="core" "readallfields"="true" "mysql.host"="127.0.0.1" "requestdistribution"="uniform" "updateproportion"="0" "scanproportion"="0" "mysql.db"="changefeed_auto_stop_1" "recordcount"="20" "mysql.user"="root" "dotransactions"="false" "threadcount"="4" ********************************************** Run finished, takes 20.139086ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1964.2, Avg(us): 3846, Min(us): 2116, Max(us): 9854, 95th(us): 10000, 99th(us): 10000 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_storage_file_large_value/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... ***************** properties ***************** "recordcount"="20" "readallfields"="true" "scanproportion"="0" "dotransactions"="false" "workload"="core" "readproportion"="0" "mysql.db"="changefeed_auto_stop_2" "insertproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" "requestdistribution"="uniform" "updateproportion"="0" "threadcount"="4" "mysql.user"="root" "mysql.port"="4000" ********************************************** Run finished, takes 17.909028ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1936.9, Avg(us): 3396, Min(us): 1960, Max(us): 7608, 95th(us): 8000, 99th(us): 8000 check diff successfully VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68896c700003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:23249, start at 2024-04-26 19:20:10.275197546 +0800 CST m=+7.561627579 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:10.291 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:10.268 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:10.268 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ***************** properties ***************** "mysql.host"="127.0.0.1" "dotransactions"="false" "readproportion"="0" "operationcount"="0" "scanproportion"="0" "workload"="core" "insertproportion"="0" "recordcount"="20" "updateproportion"="0" "mysql.user"="root" "threadcount"="4" "readallfields"="true" "mysql.port"="4000" "mysql.db"="changefeed_auto_stop_3" "requestdistribution"="uniform" ********************************************** Run finished, takes 23.067769ms INSERT - Takes(s): 0.0, Count: 20, OPS: 1495.5, Avg(us): 4393, Min(us): 2464, Max(us): 9499, 95th(us): 10000, 99th(us): 10000 [2024/04/26 19:20:10.557 +08:00] [WARN] [diff.go:745] ["rows is not equal"] [table=`test`.`itest`] [where="((TRUE) AND TRUE)"] [cost=48.590728677s] [2024/04/26 19:20:10.560 +08:00] [WARN] [diff.go:384] ["check chunk data not equal"] [chunk="{\"id\":0,\"bounds\":[],\"where\":\"((TRUE) AND TRUE)\",\"args\":null,\"state\":\"failed\"}"] ***************** properties ***************** "scanproportion"="0" "mysql.port"="4000" "recordcount"="20" "readallfields"="true" "threadcount"="4" "dotransactions"="false" "updateproportion"="0" "insertproportion"="0" "mysql.db"="changefeed_auto_stop_4" "workload"="core" "mysql.user"="root" "mysql.host"="127.0.0.1" "operationcount"="0" "requestdistribution"="uniform" "readproportion"="0" ********************************************** Run finished, takes 24.628531ms INSERT - Takes(s): 0.0, Count: 19, OPS: 1443.9, Avg(us): 4278, Min(us): 2213, Max(us): 12363, 95th(us): 13000, 99th(us): 13000 [Fri Apr 26 19:20:11 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.99069908.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc1.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data1 --cluster-id default --addr 127.0.0.1:8301 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8301; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... table ddl_attributes.finish_mark not exists for 6-th check, retry later wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68896c700003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:23249, start at 2024-04-26 19:20:10.275197546 +0800 CST m=+7.561627579 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:10.291 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:10.268 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:10.268 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68896cb4000e Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-r0wjp, pid:23290, start at 2024-04-26 19:20:10.315302523 +0800 CST m=+7.497699612 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:10.330 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:10.335 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:10.335 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/generate_column/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/generate_column/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/log/proxy.log"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash-proxy.toml"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/generate_column/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } process cdc.test already exit [Fri Apr 26 19:20:13 CST 2024] <<<<<< run test case multi_cdc_cluster success! >>>>>> table ddl_attributes.finish_mark not exists for 7-th check, retry later curl http://10.233.127.54:2379/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 19 100 19 0 0 15535 0 --:--:-- --:--:-- --:--:-- 19000 404 page not found run task successfully + echo started proxy pid: 8451 started proxy pid: 8451 + echo started proxy at port: 11949 started proxy at port: 11949 + cd /tmp/tidb_cdc_test/http_proxies ++ run_cdc_cli_tso_query 10.233.127.54 2379 + pd_host=10.233.127.54 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://10.233.127.54:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.cli.11826.out cli tso query --pd=http://10.233.127.54:2379 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8301/debug/info * About to connect() to 127.0.0.1 port 8301 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8301 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8301 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:14 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab4f020e-5b34-493a-9a79-2b5994eed48a {"id":"ab4f020e-5b34-493a-9a79-2b5994eed48a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc943 ab4f020e-5b34-493a-9a79-2b5994eed48a /tidb/cdc/default/default/upstream/7362133993870926211 {"id":7362133993870926211,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab4f020e-5b34-493a-9a79-2b5994eed48a {"id":"ab4f020e-5b34-493a-9a79-2b5994eed48a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc943 ab4f020e-5b34-493a-9a79-2b5994eed48a /tidb/cdc/default/default/upstream/7362133993870926211 {"id":7362133993870926211,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab4f020e-5b34-493a-9a79-2b5994eed48a {"id":"ab4f020e-5b34-493a-9a79-2b5994eed48a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc943 ab4f020e-5b34-493a-9a79-2b5994eed48a /tidb/cdc/default/default/upstream/7362133993870926211 {"id":7362133993870926211,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x [Fri Apr 26 19:20:14 CST 2024] <<<<<< START cdc server in changefeed_auto_stop case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/processor/pipeline/ProcessorSyncResolvedError=1*return(true);github.com/pingcap/tiflow/cdc/processor/ProcessorUpdatePositionDelaying=sleep(1000)' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.changefeed_auto_stop.99509952.out server --log-file /tmp/tidb_cdc_test/changefeed_auto_stop/cdc2.log --log-level debug --data-dir /tmp/tidb_cdc_test/changefeed_auto_stop/cdc_data2 --cluster-id default --addr 127.0.0.1:8302 --pd http://127.0.0.1:2379 + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8302; Connection refused * Closing connection 0 + res= + grep -q 'failed to get info:' + echo '' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... table ddl_attributes.finish_mark not exists for 8-th check, retry later + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.24530.out cli tso query --pd=http://127.0.0.1:2379 + set +x failed proxying s2c rpc error: code = Unavailable desc = client disconnected failed proxying s2c rpc error: code = Canceled desc = context canceled + tso='449349003353522177 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349003353522177 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349003353522177 + echo 'query start ts: 449349003353522177' query start ts: 449349003353522177 + run_cdc_server --workdir /tmp/tidb_cdc_test/http_proxies --binary cdc.test [Fri Apr 26 19:20:15 CST 2024] <<<<<< START cdc server in http_proxies case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.1186911871.out server --log-file /tmp/tidb_cdc_test/http_proxies/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/http_proxies/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to proxy 127.0.0.1 port 11949 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 11949 (#0) > GET http://127.0.0.1:8300/debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 502 Bad Gateway < Date: Fri, 26 Apr 2024 11:20:15 GMT < Content-Length: 0 < * Connection #0 to host 127.0.0.1 left intact + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled + set +x + tso='449349003711348737 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349003711348737 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:20:17 CST 2024] <<<<<< START cdc server in generate_column case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.2455424556.out server --log-file /tmp/tidb_cdc_test/generate_column/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/generate_column/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table ddl_attributes.finish_mark not exists for 9-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8302/debug/info * About to connect() to 127.0.0.1 port 8302 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8302 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8302 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:17 GMT < Content-Length: 867 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab4f020e-5b34-493a-9a79-2b5994eed48a {"id":"ab4f020e-5b34-493a-9a79-2b5994eed48a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d2b6485b-3602-48ff-9cde-e95b1f99c7db {"id":"d2b6485b-3602-48ff-9cde-e95b1f99c7db","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc943 ab4f020e-5b34-493a-9a79-2b5994eed48a /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc96a d2b6485b-3602-48ff-9cde-e95b1f99c7db /tidb/cdc/default/default/upstream/7362133993870926211 {"id":7362133993870926211,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab4f020e-5b34-493a-9a79-2b5994eed48a {"id":"ab4f020e-5b34-493a-9a79-2b5994eed48a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d2b6485b-3602-48ff-9cde-e95b1f99c7db {"id":"d2b6485b-3602-48ff-9cde-e95b1f99c7db","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc943 ab4f020e-5b34-493a-9a79-2b5994eed48a /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc96a d2b6485b-3602-48ff-9cde-e95b1f99c7db /tidb/cdc/default/default/upstream/7362133993870926211 {"id":7362133993870926211,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ab4f020e-5b34-493a-9a79-2b5994eed48a {"id":"ab4f020e-5b34-493a-9a79-2b5994eed48a","address":"127.0.0.1:8301","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/capture/d2b6485b-3602-48ff-9cde-e95b1f99c7db {"id":"d2b6485b-3602-48ff-9cde-e95b1f99c7db","address":"127.0.0.1:8302","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc943 ab4f020e-5b34-493a-9a79-2b5994eed48a /tidb/cdc/default/__cdc_meta__/owner/22318f1a221cc96a d2b6485b-3602-48ff-9cde-e95b1f99c7db /tidb/cdc/default/default/upstream/7362133993870926211 {"id":7362133993870926211,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x check_changefeed_state http://127.0.0.1:2379 38ce4bb8-268e-4199-a455-42e53b08c439 normal null + endpoints=http://127.0.0.1:2379 + changefeed_id=38ce4bb8-268e-4199-a455-42e53b08c439 + expected_state=normal + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c 38ce4bb8-268e-4199-a455-42e53b08c439 -s + info='{ "upstream_id": 7362133993870926211, "namespace": "default", "id": "38ce4bb8-268e-4199-a455-42e53b08c439", "state": "normal", "checkpoint_tso": 449349002244128769, "checkpoint_time": "2024-04-26 19:20:10.172", "error": null }' + echo '{ "upstream_id": 7362133993870926211, "namespace": "default", "id": "38ce4bb8-268e-4199-a455-42e53b08c439", "state": "normal", "checkpoint_tso": 449349002244128769, "checkpoint_time": "2024-04-26 19:20:10.172", "error": null }' { "upstream_id": 7362133993870926211, "namespace": "default", "id": "38ce4bb8-268e-4199-a455-42e53b08c439", "state": "normal", "checkpoint_tso": 449349002244128769, "checkpoint_time": "2024-04-26 19:20:10.172", "error": null } ++ echo '{' '"upstream_id":' 7362133993870926211, '"namespace":' '"default",' '"id":' '"38ce4bb8-268e-4199-a455-42e53b08c439",' '"state":' '"normal",' '"checkpoint_tso":' 449349002244128769, '"checkpoint_time":' '"2024-04-26' '19:20:10.172",' '"error":' null '}' ++ jq -r .state + state=normal + [[ ! normal == \n\o\r\m\a\l ]] ++ jq -r .error.message ++ echo '{' '"upstream_id":' 7362133993870926211, '"namespace":' '"default",' '"id":' '"38ce4bb8-268e-4199-a455-42e53b08c439",' '"state":' '"normal",' '"checkpoint_tso":' 449349002244128769, '"checkpoint_time":' '"2024-04-26' '19:20:10.172",' '"error":' null '}' + message=null + [[ ! null =~ null ]] run task successfully table changefeed_auto_stop_1.usertable not exists for 1-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to proxy 127.0.0.1 port 11949 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 11949 (#0) > GET http://127.0.0.1:8300/debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 OK < Content-Length: 617 < Content-Type: text/plain; charset=utf-8 < Date: Fri, 26 Apr 2024 11:20:18 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c2061c2c-ca3e-4bec-834f-eaad00b16e8a {"id":"c2061c2c-ca3e-4bec-834f-eaad00b16e8a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/6bf78f1a21f12d10 c2061c2c-ca3e-4bec-834f-eaad00b16e8a /tidb/cdc/default/default/upstream/7362133936332695150 {"id":7362133936332695150,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.127.54:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c2061c2c-ca3e-4bec-834f-eaad00b16e8a {"id":"c2061c2c-ca3e-4bec-834f-eaad00b16e8a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/6bf78f1a21f12d10 c2061c2c-ca3e-4bec-834f-eaad00b16e8a /tidb/cdc/default/default/upstream/7362133936332695150 {"id":7362133936332695150,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.127.54:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c2061c2c-ca3e-4bec-834f-eaad00b16e8a {"id":"c2061c2c-ca3e-4bec-834f-eaad00b16e8a","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/6bf78f1a21f12d10 c2061c2c-ca3e-4bec-834f-eaad00b16e8a /tidb/cdc/default/default/upstream/7362133936332695150 {"id":7362133936332695150,"pd-endpoints":"http://127.0.0.1:2379,http://10.233.127.54:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + echo started cdc server successfully started cdc server successfully + SINK_URI=blackhole:/// + run_cdc_cli changefeed create --start-ts=449349003353522177 --sink-uri=blackhole:/// + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.http_proxies.cli.11924.out cli changefeed create --start-ts=449349003353522177 --sink-uri=blackhole:/// \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) failed proxying s2c rpc error: code = Canceled desc = context canceled Create changefeed successfully! ID: d23d7fb0-dca4-4c05-8e30-159f7c01ce0d Info: {"upstream_id":7362133936332695150,"namespace":"default","id":"d23d7fb0-dca4-4c05-8e30-159f7c01ce0d","sink_uri":"blackhole:///","create_time":"2024-04-26T19:20:19.331473434+08:00","start_ts":449349003353522177,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349003353522177,"checkpoint_ts":449349003353522177,"checkpoint_time":"2024-04-26 19:20:14.404"} PASS [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } coverage: 2.5% of statements in github.com/pingcap/tiflow/... [Pipeline] // timeout [Pipeline] } Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // stage [Pipeline] } table ddl_attributes.finish_mark exists check diff successfully [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } table changefeed_auto_stop_1.usertable exists table changefeed_auto_stop_2.usertable not exists for 1-th check, retry later [Pipeline] // podTemplate [Pipeline] } wait process cdc.test exit for 1-th time... [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } + set +x + sleep 5 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:20 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e43ccad-ba67-4c37-9165-b2795faa89fb {"id":"9e43ccad-ba67-4c37-9165-b2795faa89fb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a222590f9 9e43ccad-ba67-4c37-9165-b2795faa89fb /tidb/cdc/default/default/upstream/7362134007148674392 {"id":7362134007148674392,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e43ccad-ba67-4c37-9165-b2795faa89fb {"id":"9e43ccad-ba67-4c37-9165-b2795faa89fb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a222590f9 9e43ccad-ba67-4c37-9165-b2795faa89fb /tidb/cdc/default/default/upstream/7362134007148674392 {"id":7362134007148674392,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/9e43ccad-ba67-4c37-9165-b2795faa89fb {"id":"9e43ccad-ba67-4c37-9165-b2795faa89fb","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a222590f9 9e43ccad-ba67-4c37-9165-b2795faa89fb /tidb/cdc/default/default/upstream/7362134007148674392 {"id":7362134007148674392,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.generate_column.cli.24603.out cli changefeed create --start-ts=449349003711348737 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ wait process cdc.test exit for 2-th time... Create changefeed successfully! ID: 92d1b532-8d36-4a9f-af42-043e861b6502 Info: {"upstream_id":7362134007148674392,"namespace":"default","id":"92d1b532-8d36-4a9f-af42-043e861b6502","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:20:21.15403894+08:00","start_ts":449349003711348737,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349003711348737,"checkpoint_ts":449349003711348737,"checkpoint_time":"2024-04-26 19:20:15.769"} PASS cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit coverage: 2.4% of statements in github.com/pingcap/tiflow/... [Fri Apr 26 19:20:21 CST 2024] <<<<<< run test case ddl_attributes success! >>>>>> ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table changefeed_auto_stop_2.usertable not exists for 2-th check, retry later + set +x \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } table generate_column.t not exists for 1-th check, retry later [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } table changefeed_auto_stop_2.usertable exists table changefeed_auto_stop_3.usertable not exists for 1-th check, retry later [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table generate_column.t exists table generate_column.t1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + check + services=($(cat $WORK_DIR/test_proxy.log | xargs -L1 dirname | sort | uniq)) ++ cat /tmp/tidb_cdc_test/http_proxies/test_proxy.log ++ xargs -L1 dirname ++ sort ++ uniq xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option + service_type_count=5 + echo 'captured services: ' captured services: + echo . /grpc.health.v1.Health /pdpb.PD /resource_manager.ResourceManager '[2024/04' . /grpc.health.v1.Health /pdpb.PD /resource_manager.ResourceManager [2024/04 + '[' 5 -ge 2 ']' + check_logs /tmp/tidb_cdc_test/http_proxies ++ date + echo '[Fri Apr 26 19:20:25 CST 2024] <<<<<< run test case http_proxies success! >>>>>>' [Fri Apr 26 19:20:25 CST 2024] <<<<<< run test case http_proxies success! >>>>>> + stop_tidb_cluster failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Unavailable desc = client disconnected failed proxying s2c rpc error: code = Unavailable desc = client disconnected failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Canceled desc = context canceled failed proxying s2c rpc error: code = Unavailable desc = client disconnected failed proxying s2c rpc error: code = Canceled desc = context canceled [2024/04/26 19:20:24.374 +08:00] [INFO] [dailytest.go:68] ["test pass!!!"] wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:20:26 CST 2024] <<<<<< run test case cdc success! >>>>>> table changefeed_auto_stop_3.usertable exists table changefeed_auto_stop_4.usertable not exists for 1-th check, retry later table generate_column.t1 exists check diff failed 1-th time, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688a539c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:12406, start at 2024-04-26 19:20:25.075797528 +0800 CST m=+5.576430123 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:25.088 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:25.063 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:25.063 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688a539c0009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:12406, start at 2024-04-26 19:20:25.075797528 +0800 CST m=+5.576430123 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:25.088 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:25.063 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:25.063 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688a54340003 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:12458, start at 2024-04-26 19:20:25.102333744 +0800 CST m=+5.524714167 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:25.110 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:25.101 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:25.101 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash-proxy.toml"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } valid ~~~ running cdc Failed to start cdc, the usage tips should be printed 1st test case cdc_server_tips success! try an INVALID cdc server command [Fri Apr 26 19:20:25 CST 2024] <<<<<< START cdc server in cdc_server_tips case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ true != \n\o ]] + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.cdc_server_tips.2483724839.out server --log-file /tmp/tidb_cdc_test/cdc_server_tips/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/cdc_server_tips/cdc_data --cluster-id default --pd None table changefeed_auto_stop_4.usertable exists check diff failed 1-th time, retry later check diff successfully wait process cdc.test exit for 1-th time... [Fri Apr 26 19:20:29 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_file_large_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file_large_value.1368113683.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 2-th time... check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 3-th time... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } wait process cdc.test exit for 2-th time... [Pipeline] // timeout [Pipeline] } [Pipeline] // stage cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:20:31 CST 2024] <<<<<< run test case generate_column success! >>>>>> [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate wait process cdc.test exit for 3-th time... [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:20:32 CST 2024] <<<<<< run test case changefeed_auto_stop success! >>>>>> + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:33 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6aa3f97-2da6-48f7-8ba8-a904ede4d92c {"id":"e6aa3f97-2da6-48f7-8ba8-a904ede4d92c","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a226b12f6 e6aa3f97-2da6-48f7-8ba8-a904ede4d92c /tidb/cdc/default/default/upstream/7362134070045830248 {"id":7362134070045830248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6aa3f97-2da6-48f7-8ba8-a904ede4d92c {"id":"e6aa3f97-2da6-48f7-8ba8-a904ede4d92c","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a226b12f6 e6aa3f97-2da6-48f7-8ba8-a904ede4d92c /tidb/cdc/default/default/upstream/7362134070045830248 {"id":7362134070045830248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/e6aa3f97-2da6-48f7-8ba8-a904ede4d92c {"id":"e6aa3f97-2da6-48f7-8ba8-a904ede4d92c","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a226b12f6 e6aa3f97-2da6-48f7-8ba8-a904ede4d92c /tidb/cdc/default/default/upstream/7362134070045830248 {"id":7362134070045830248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x ***************** properties ***************** "mysql.port"="4000" "fieldcount"="1000" "operationcount"="0" "workload"="core" "updateproportion"="0" "scanproportion"="0" "threadcount"="10" "mysql.host"="127.0.0.1" "dotransactions"="false" "insertproportion"="0" "fieldlength"="15000" "requestdistribution"="uniform" "readallfields"="true" "mysql.user"="root" "readproportion"="0" "recordcount"="100" "fieldlengthdistribution"="constant" "mysql.db"="consistent_replicate_storage_file_large_value" ********************************************** + stop_proxy + kill 8451 =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/sequence/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/sequence Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/split_region/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... [Pipeline] } [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } INSERT_ERROR - Takes(s): 8.8, Count: 80, OPS: 9.1, Avg(us): 414049, Min(us): 235438, Max(us): 678884, 95th(us): 616000, 99th(us): 679000 [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } invalid ~~~ running cdc Failed to start cdc, the usage tips should be printed 2nd test case cdc_server_tips success! [Fri Apr 26 19:20:45 CST 2024] <<<<<< run all test cases cdc_server_tips success! >>>>>> Run finished, takes 11.525688381s INSERT_ERROR - Takes(s): 10.2, Count: 100, OPS: 9.8, Avg(us): 387494, Min(us): 169500, Max(us): 678884, 95th(us): 613000, 99th(us): 643000 start tidb cluster in /tmp/tidb_cdc_test/split_region Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file_large_value.usertable exists table consistent_replicate_storage_file_large_value.check1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file_large_value.check1 exists table consistent_replicate_storage_file_large_value.t2 not exists for 1-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688bdb240012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:12906, start at 2024-04-26 19:20:50.143610332 +0800 CST m=+5.249026071 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:50.150 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:50.121 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:50.121 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688bdb240012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:12906, start at 2024-04-26 19:20:50.143610332 +0800 CST m=+5.249026071 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:50.150 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:50.121 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:50.121 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688bdb080015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-c25k0, pid:12984, start at 2024-04-26 19:20:50.150312134 +0800 CST m=+5.202503475 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:50.159 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:50.165 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:50.165 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/sequence/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/sequence/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/sequence/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/sequence/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/sequence/tiflash/db/proxy"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file_large_value.t2 not exists for 2-th check, retry later [Fri Apr 26 19:20:53 CST 2024] <<<<<< START cdc server in sequence case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sequence.1424414246.out server --log-file /tmp/tidb_cdc_test/sequence/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/sequence/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_storage_file_large_value.t2 not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_file_large_value.t2 not exists for 4-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:20:56 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06 {"id":"cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a22d048f4 cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06 /tidb/cdc/default/default/upstream/7362134180986970439 {"id":7362134180986970439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06 {"id":"cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a22d048f4 cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06 /tidb/cdc/default/default/upstream/7362134180986970439 {"id":7362134180986970439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06 {"id":"cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a22d048f4 cf8c933d-6b7c-4e4f-bfc1-e2bd2dd1de06 /tidb/cdc/default/default/upstream/7362134180986970439 {"id":7362134180986970439,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.sequence.cli.14294.out cli changefeed create --start-ts=449349013581594625 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --config /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/sequence/conf/force_replicate.toml [2024/04/26 19:20:51.415 +08:00] [INFO] [main.go:86] ["running ddl test: 0 createDropSchemaDDL"] [2024/04/26 19:20:53.011 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/04/26 19:20:53.045 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/04/26 19:20:54.534 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/04/26 19:20:55.072 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/04/26 19:20:55.074 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/04/26 19:20:55.088 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: bb316519-1849-4952-adfb-ddc5eb5a7158 Info: {"upstream_id":7362134180986970439,"namespace":"default","id":"bb316519-1849-4952-adfb-ddc5eb5a7158","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:20:56.910279369+08:00","start_ts":449349013581594625,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349013581594625,"checkpoint_ts":449349013581594625,"checkpoint_time":"2024-04-26 19:20:53.421"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/event_filter/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... table consistent_replicate_storage_file_large_value.t2 not exists for 5-th check, retry later + set +x table sequence_test.t1 not exists for 1-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688c52400014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:22976, start at 2024-04-26 19:20:57.779229647 +0800 CST m=+5.290865934 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:57.785 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:57.744 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:57.744 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) start tidb cluster in /tmp/tidb_cdc_test/event_filter Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table consistent_replicate_storage_file_large_value.t2 not exists for 6-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688c52400014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:22976, start at 2024-04-26 19:20:57.779229647 +0800 CST m=+5.290865934 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:57.785 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:57.744 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:57.744 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688c52600015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-qgmtf, pid:23042, start at 2024-04-26 19:20:57.798488127 +0800 CST m=+5.251499320 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:22:57.806 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:20:57.802 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:10:57.802 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/split_region/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/split_region/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/split_region/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/db/proxy"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/split_region/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table sequence_test.t1 exists check table exists success check diff failed 1-th time, retry later [2024/04/26 19:21:01.559 +08:00] [INFO] [main.go:86] ["running ddl test: 1 truncateDDL"] [2024/04/26 19:21:02.129 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/04/26 19:21:02.232 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] table consistent_replicate_storage_file_large_value.t2 exists check diff successfully + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.24296.out cli tso query --pd=http://127.0.0.1:2379 [2024/04/26 19:21:02.656 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/04/26 19:21:02.666 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/04/26 19:21:02.835 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/04/26 19:21:02.842 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] check diff successfully ERROR 1146 (42S02) at line 1: Table 'sequence_test.seq0' doesn't exist wait process cdc.test exit for 1-th time... [2024/04/26 19:21:03.167 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] table sequence_test.mark_table not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... [2024/04/26 19:21:03.478 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] [2024/04/26 19:21:03.751 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/04/26 19:21:03.772 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:21:03 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_file_large_value case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + etcd_info_msg='etcd info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_file_large_value.1394513947.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:21:04.121 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] [2024/04/26 19:21:04.175 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/04/26 19:21:04.296 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] check_changefeed_state http://127.0.0.1:2379 b048cbfb-c0b1-48ca-bc83-fbf572d91127 finished null + endpoints=http://127.0.0.1:2379 + changefeed_id=b048cbfb-c0b1-48ca-bc83-fbf572d91127 + expected_state=finished + error_msg=null + tls_dir=null + [[ http://127.0.0.1:2379 =~ https ]] ++ cdc cli changefeed query --pd=http://127.0.0.1:2379 -c b048cbfb-c0b1-48ca-bc83-fbf572d91127 -s + info='{ "upstream_id": 7362133778593397932, "namespace": "default", "id": "b048cbfb-c0b1-48ca-bc83-fbf572d91127", "state": "finished", "checkpoint_tso": 449349013451571202, "checkpoint_time": "2024-04-26 19:20:52.925", "error": null }' + echo '{ "upstream_id": 7362133778593397932, "namespace": "default", "id": "b048cbfb-c0b1-48ca-bc83-fbf572d91127", "state": "finished", "checkpoint_tso": 449349013451571202, "checkpoint_time": "2024-04-26 19:20:52.925", "error": null }' { "upstream_id": 7362133778593397932, "namespace": "default", "id": "b048cbfb-c0b1-48ca-bc83-fbf572d91127", "state": "finished", "checkpoint_tso": 449349013451571202, "checkpoint_time": "2024-04-26 19:20:52.925", "error": null } ++ echo '{' '"upstream_id":' 7362133778593397932, '"namespace":' '"default",' '"id":' '"b048cbfb-c0b1-48ca-bc83-fbf572d91127",' '"state":' '"finished",' '"checkpoint_tso":' 449349013451571202, '"checkpoint_time":' '"2024-04-26' '19:20:52.925",' '"error":' null '}' ++ jq -r .state + state=finished + [[ ! finished == \f\i\n\i\s\h\e\d ]] ++ echo '{' '"upstream_id":' 7362133778593397932, '"namespace":' '"default",' '"id":' '"b048cbfb-c0b1-48ca-bc83-fbf572d91127",' '"state":' '"finished",' '"checkpoint_tso":' 449349013451571202, '"checkpoint_time":' '"2024-04-26' '19:20:52.925",' '"error":' null '}' ++ jq -r .error.message + message=null + [[ ! null =~ null ]] run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:20:59 CST 2024] <<<<<< run test case changefeed_finish success! >>>>>> + set +x + tso='449349016117313537 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349016117313537 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x [Fri Apr 26 19:21:04 CST 2024] <<<<<< START cdc server in split_region case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS= + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.2433524337.out server --log-file /tmp/tidb_cdc_test/split_region/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/split_region/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 [2024/04/26 19:21:04.680 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] [2024/04/26 19:21:04.870 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/04/26 19:21:04.952 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/04/26 19:21:05.365 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] [2024/04/26 19:21:05.422 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] [2024/04/26 19:21:05.465 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] table sequence_test.mark_table not exists for 2-th check, retry later [2024/04/26 19:21:05.874 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] [2024/04/26 19:21:05.956 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] Starting Upstream TiDB... [2024/04/26 19:21:06.144 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:21:06.476 +08:00] [INFO] [main.go:220] ["0 insert success: 800"] + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:21:07 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {UpstreamID:7362134070045830248 Namespace:default ID:5e19838f-c3a4-4100-a3de-c6159f7b1b99 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:20:33.241648698 +0800 CST StartTs:449349008257974274 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002b36ea0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349008284188680} {CheckpointTs:449349012045955096 MinTableBarrierTs:449349017052905474 AdminJobType:noop} span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d5462b9-e880-4159-96ee-f3acedb8626e {"id":"0d5462b9-e880-4159-96ee-f3acedb8626e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a226b13da 0d5462b9-e880-4159-96ee-f3acedb8626e /tidb/cdc/default/default/changefeed/info/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"upstream-id":7362134070045830248,"namespace":"default","changefeed-id":"5e19838f-c3a4-4100-a3de-c6159f7b1b99","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:20:33.241648698+08:00","start-ts":449349008257974274,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349008284188680} /tidb/cdc/default/default/changefeed/status/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"checkpoint-ts":449349012045955096,"min-table-barrier-ts":449349017052905474,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d5462b9-e880-4159-96ee-f3acedb8626e/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362134070045830248 {"id":7362134070045830248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {UpstreamID:7362134070045830248 Namespace:default ID:5e19838f-c3a4-4100-a3de-c6159f7b1b99 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:20:33.241648698 +0800 CST StartTs:449349008257974274 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002b36ea0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349008284188680} {CheckpointTs:449349012045955096 MinTableBarrierTs:449349017052905474 AdminJobType:noop} span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d5462b9-e880-4159-96ee-f3acedb8626e {"id":"0d5462b9-e880-4159-96ee-f3acedb8626e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a226b13da 0d5462b9-e880-4159-96ee-f3acedb8626e /tidb/cdc/default/default/changefeed/info/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"upstream-id":7362134070045830248,"namespace":"default","changefeed-id":"5e19838f-c3a4-4100-a3de-c6159f7b1b99","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:20:33.241648698+08:00","start-ts":449349008257974274,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349008284188680} /tidb/cdc/default/default/changefeed/status/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"checkpoint-ts":449349012045955096,"min-table-barrier-ts":449349017052905474,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d5462b9-e880-4159-96ee-f3acedb8626e/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362134070045830248 {"id":7362134070045830248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {UpstreamID:7362134070045830248 Namespace:default ID:5e19838f-c3a4-4100-a3de-c6159f7b1b99 SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:20:33.241648698 +0800 CST StartTs:449349008257974274 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc002b36ea0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349008284188680} {CheckpointTs:449349012045955096 MinTableBarrierTs:449349017052905474 AdminJobType:noop} span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449349012045955096, checkpointTs: 449349012045955096, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0d5462b9-e880-4159-96ee-f3acedb8626e {"id":"0d5462b9-e880-4159-96ee-f3acedb8626e","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a226b13da 0d5462b9-e880-4159-96ee-f3acedb8626e /tidb/cdc/default/default/changefeed/info/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"upstream-id":7362134070045830248,"namespace":"default","changefeed-id":"5e19838f-c3a4-4100-a3de-c6159f7b1b99","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:20:33.241648698+08:00","start-ts":449349008257974274,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349008284188680} /tidb/cdc/default/default/changefeed/status/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"checkpoint-ts":449349012045955096,"min-table-barrier-ts":449349017052905474,"admin-job-type":0} /tidb/cdc/default/default/task/position/0d5462b9-e880-4159-96ee-f3acedb8626e/5e19838f-c3a4-4100-a3de-c6159f7b1b99 {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362134070045830248 {"id":7362134070045830248,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x table sequence_test.mark_table not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:21:07 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2fd8aef-f713-4f92-ba47-fb971d287805 {"id":"b2fd8aef-f713-4f92-ba47-fb971d287805","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a22ea2711 b2fd8aef-f713-4f92-ba47-fb971d287805 /tidb/cdc/default/default/upstream/7362134220800914588 {"id":7362134220800914588,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2fd8aef-f713-4f92-ba47-fb971d287805 {"id":"b2fd8aef-f713-4f92-ba47-fb971d287805","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a22ea2711 b2fd8aef-f713-4f92-ba47-fb971d287805 /tidb/cdc/default/default/upstream/7362134220800914588 {"id":7362134220800914588,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b2fd8aef-f713-4f92-ba47-fb971d287805 {"id":"b2fd8aef-f713-4f92-ba47-fb971d287805","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a22ea2711 b2fd8aef-f713-4f92-ba47-fb971d287805 /tidb/cdc/default/default/upstream/7362134220800914588 {"id":7362134220800914588,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.split_region.cli.24381.out cli changefeed create --start-ts=449349016117313537 --sink-uri=mysql://normal:123456@127.0.0.1:3306/ Create changefeed successfully! ID: b8f8e56e-72c5-4e45-bf04-98fb9c50552d Info: {"upstream_id":7362134220800914588,"namespace":"default","id":"b8f8e56e-72c5-4e45-bf04-98fb9c50552d","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:21:08.270230475+08:00","start_ts":449349016117313537,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349016117313537,"checkpoint_ts":449349016117313537,"checkpoint_time":"2024-04-26 19:21:03.094"} PASS coverage: 2.4% of statements in github.com/pingcap/tiflow/... table sequence_test.mark_table exists =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/force_replicate_table/run.sh using Sink-Type: mysql... <<================= wait process cdc.test exit for 1-th time... The 1 times to try to start tidb cluster... + set +x table split_region.test1 not exists for 1-th check, retry later wait process cdc.test exit for 2-th time... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:21:10 CST 2024] <<<<<< run test case sequence success! >>>>>> start tidb cluster in /tmp/tidb_cdc_test/force_replicate_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... table split_region.test1 exists table split_region.test2 not exists for 1-th check, retry later Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688d2880000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:25785, start at 2024-04-26 19:21:11.46907105 +0800 CST m=+5.306580293 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:11.477 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:11.456 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:11.456 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:21:11.873 +08:00] [INFO] [main.go:86] ["running ddl test: 2 addDropColumnDDL"] [2024/04/26 19:21:12.462 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/04/26 19:21:12.463 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/04/26 19:21:12.946 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/04/26 19:21:12.950 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/04/26 19:21:12.952 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/04/26 19:21:12.956 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/04/26 19:21:13.473 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/04/26 19:21:13.491 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] table split_region.test2 exists [2024/04/26 19:21:13.970 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] [2024/04/26 19:21:13.975 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/04/26 19:21:13.980 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] check diff successfully [2024/04/26 19:21:13.983 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/04/26 19:21:14.476 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] [2024/04/26 19:21:14.479 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688d2880000c Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:25785, start at 2024-04-26 19:21:11.46907105 +0800 CST m=+5.306580293 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:11.477 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:11.456 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:11.456 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688d2a000016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-djcjh, pid:25859, start at 2024-04-26 19:21:11.5974467 +0800 CST m=+5.376444474 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:11.605 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:11.602 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:11.602 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/event_filter/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/event_filter/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/event_filter/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [2024/04/26 19:21:14.965 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] [2024/04/26 19:21:14.969 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] [2024/04/26 19:21:15.471 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/04/26 19:21:15.476 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/04/26 19:21:15.494 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release [2024/04/26 19:21:15.953 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] [2024/04/26 19:21:15.979 +08:00] [INFO] [main.go:220] ["0 insert success: 800"] [2024/04/26 19:21:15.986 +08:00] [INFO] [main.go:234] ["0 delete success: 400"] [2024/04/26 19:21:16.440 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] [2024/04/26 19:21:16.445 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] [2024/04/26 19:21:16.466 +08:00] [INFO] [main.go:220] ["0 insert success: 900"] [Fri Apr 26 19:21:16 CST 2024] <<<<<< START cdc server in event_filter case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.2715927161.out server --log-file /tmp/tidb_cdc_test/event_filter/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/event_filter/cdc_data --cluster-id default --addr 127.0.0.1:8300 --pd http://127.0.0.1:2379 + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:21:19 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1774d9f-8a5c-45c0-b78c-654581c75a63 {"id":"d1774d9f-8a5c-45c0-b78c-654581c75a63","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a231d91f2 d1774d9f-8a5c-45c0-b78c-654581c75a63 /tidb/cdc/default/default/upstream/7362134280268437816 {"id":7362134280268437816,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1774d9f-8a5c-45c0-b78c-654581c75a63 {"id":"d1774d9f-8a5c-45c0-b78c-654581c75a63","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a231d91f2 d1774d9f-8a5c-45c0-b78c-654581c75a63 /tidb/cdc/default/default/upstream/7362134280268437816 {"id":7362134280268437816,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d1774d9f-8a5c-45c0-b78c-654581c75a63 {"id":"d1774d9f-8a5c-45c0-b78c-654581c75a63","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a231d91f2 d1774d9f-8a5c-45c0-b78c-654581c75a63 /tidb/cdc/default/default/upstream/7362134280268437816 {"id":7362134280268437816,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.event_filter.cli.27211.out cli changefeed create --sink-uri=mysql://normal:123456@127.0.0.1:3306/ --server=127.0.0.1:8300 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/event_filter/conf/cf.toml ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) Create changefeed successfully! ID: 03ed905f-21cc-4afc-81f9-e703ce99de90 Info: {"upstream_id":7362134280268437816,"namespace":"default","id":"03ed905f-21cc-4afc-81f9-e703ce99de90","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/","create_time":"2024-04-26T19:21:19.989396548+08:00","start_ts":449349020511371268,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["event_filter.*"],"event_filters":[{"matcher":["event_filter.t1"],"ignore_event":["drop table","delete"],"ignore_sql":null,"ignore_insert_value_expr":"id = 2 or city = 'tokyo'","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_truncate"],"ignore_event":["truncate table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""},{"matcher":["event_filter.t_alter"],"ignore_event":["alter table"],"ignore_sql":null,"ignore_insert_value_expr":"","ignore_update_new_value_expr":"","ignore_update_old_value_expr":"","ignore_delete_value_expr":""}]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349020511371268,"checkpoint_ts":449349020511371268,"checkpoint_time":"2024-04-26 19:21:19.856"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } + set +x [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table event_filter.t1 does not exists table event_filter.t1 not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:21:22.288 +08:00] [INFO] [main.go:86] ["running ddl test: 3 addDropColumnDDL2"] [2024/04/26 19:21:22.859 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/04/26 19:21:22.863 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] [2024/04/26 19:21:23.374 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/04/26 19:21:23.385 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/04/26 19:21:23.392 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/04/26 19:21:23.399 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/04/26 19:21:23.930 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] [2024/04/26 19:21:23.931 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688de2840012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:6535, start at 2024-04-26 19:21:23.382587928 +0800 CST m=+5.482094211 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:23.389 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:23.361 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:23.361 +0800 All versions after safe point can be accessed. (DO NOT EDIT) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) [2024/04/26 19:21:24.402 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/04/26 19:21:24.407 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/04/26 19:21:24.418 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] [2024/04/26 19:21:24.423 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] table event_filter.t1 exists table event_filter.t_normal not exists for 1-th check, retry later [2024/04/26 19:21:24.866 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] [2024/04/26 19:21:24.866 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] [2024/04/26 19:21:25.375 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/04/26 19:21:25.384 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/04/26 19:21:25.389 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] [2024/04/26 19:21:25.400 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] check diff failed 1-th time, retry later [2024/04/26 19:21:25.983 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] [2024/04/26 19:21:26.029 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688de2840012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:6535, start at 2024-04-26 19:21:23.382587928 +0800 CST m=+5.482094211 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:23.389 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:23.361 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:23.361 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c688de3280015 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-tcv4c, pid:6617, start at 2024-04-26 19:21:23.448682538 +0800 CST m=+5.477488953 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:23:23.456 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:21:23.453 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:11:23.453 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/force_replicate_table/tiflash/log/error.log arg matches is ArgMatches { args: {"config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/db/proxy"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/force_replicate_table/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table event_filter.t_normal exists table event_filter.t_truncate not exists for 1-th check, retry later [2024/04/26 19:21:26.453 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] [2024/04/26 19:21:26.459 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] [2024/04/26 19:21:26.564 +08:00] [INFO] [main.go:220] ["0 insert success: 800"] [2024/04/26 19:21:26.568 +08:00] [INFO] [main.go:234] ["0 delete success: 400"] [2024/04/26 19:21:26.959 +08:00] [INFO] [main.go:220] ["1 insert success: 900"] [2024/04/26 19:21:27.130 +08:00] [INFO] [main.go:220] ["0 insert success: 900"] table event_filter.t_truncate exists table event_filter.t_alter not exists for 1-th check, retry later check diff successfully [Fri Apr 26 19:21:28 CST 2024] <<<<<< START cdc server in force_replicate_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.force_replicate_table.80008002.out server --log-file /tmp/tidb_cdc_test/force_replicate_table/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/force_replicate_table/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 wait process cdc.test exit for 1-th time... check_redo_resolved_ts 5e19838f-c3a4-4100-a3de-c6159f7b1b99 449349022741430276 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/5e19838f-c3a4-4100-a3de-c6159f7b1b99/meta global resolved ts 449349022033641476 not forward to 449349022741430276 run task failed 1-th time, retry later wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:21:29 CST 2024] <<<<<< run test case split_region success! >>>>>> table event_filter.t_alter exists check_redo_resolved_ts 5e19838f-c3a4-4100-a3de-c6159f7b1b99 449349022741430276 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/5e19838f-c3a4-4100-a3de-c6159f7b1b99/meta global resolved ts 449349022557929476 not forward to 449349022741430276 run task failed 2-th time, retry later table event_filter.finish_mark exists check diff failed 1-th time, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:21:31 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f9c54447-247c-4311-be2e-19ab626585be {"id":"f9c54447-247c-4311-be2e-19ab626585be","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a234b7cf2 f9c54447-247c-4311-be2e-19ab626585be /tidb/cdc/default/default/upstream/7362134327460122634 {"id":7362134327460122634,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f9c54447-247c-4311-be2e-19ab626585be {"id":"f9c54447-247c-4311-be2e-19ab626585be","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a234b7cf2 f9c54447-247c-4311-be2e-19ab626585be /tidb/cdc/default/default/upstream/7362134327460122634 {"id":7362134327460122634,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/f9c54447-247c-4311-be2e-19ab626585be {"id":"f9c54447-247c-4311-be2e-19ab626585be","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a234b7cf2 f9c54447-247c-4311-be2e-19ab626585be /tidb/cdc/default/default/upstream/7362134327460122634 {"id":7362134327460122634,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x Create changefeed successfully! ID: de5f305f-6731-4e12-af7a-7dba0a4c2146 Info: {"upstream_id":7362134327460122634,"namespace":"default","id":"de5f305f-6731-4e12-af7a-7dba0a4c2146","sink_uri":"mysql://normal:xxxxx@127.0.0.1:3306/?safe-mode=true","create_time":"2024-04-26T19:21:31.595550625+08:00","start_ts":449349022728060929,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":true,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":300,"checkpoint_interval":15}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349022728060929,"checkpoint_ts":449349022728060929,"checkpoint_time":"2024-04-26 19:21:28.312"} [2024/04/26 19:21:32.431 +08:00] [INFO] [main.go:86] ["running ddl test: 4 modifyColumnDDL"] [2024/04/26 19:21:32.977 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/04/26 19:21:33.000 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] check diff failed 2-th time, retry later table force_replicate_table.t0 exists table force_replicate_table.t1 not exists for 1-th check, retry later [2024/04/26 19:21:33.492 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] [2024/04/26 19:21:33.500 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/04/26 19:21:33.544 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/04/26 19:21:33.554 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/04/26 19:21:34.041 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] [2024/04/26 19:21:34.048 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/04/26 19:21:34.537 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/04/26 19:21:34.542 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/04/26 19:21:34.544 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/04/26 19:21:34.546 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] check_redo_resolved_ts 5e19838f-c3a4-4100-a3de-c6159f7b1b99 449349022741430276 file:///tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/redo /tmp/tidb_cdc_test/consistent_replicate_storage_file_large_value/cdc_data/redo/5e19838f-c3a4-4100-a3de-c6159f7b1b99/meta run task successfully [2024/04/26 19:21:34.970 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] [2024/04/26 19:21:35.013 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] wait process cdc.test exit for 1-th time... table force_replicate_table.t1 not exists for 2-th check, retry later [2024/04/26 19:21:35.488 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/04/26 19:21:35.492 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] [2024/04/26 19:21:35.521 +08:00] [INFO] [main.go:234] ["0 delete success: 300"] [2024/04/26 19:21:35.524 +08:00] [INFO] [main.go:220] ["0 insert success: 600"] wait process cdc.test exit for 2-th time... check diff successfully [2024/04/26 19:21:35.979 +08:00] [INFO] [main.go:220] ["1 insert success: 700"] [2024/04/26 19:21:36.010 +08:00] [INFO] [main.go:220] ["0 insert success: 700"] wait process cdc.test exit for 1-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [2024/04/26 19:21:36.309 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/04/26 19:21:36.309 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/04/26 19:21:36.309 +08:00] [INFO] [redo.go:163] ["apply redo log starts"] [checkpointTs=449349017341001734] [resolvedTs=449349023606505477] [2024/04/26 19:21:36.310 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=863.212ยตs] [2024/04/26 19:21:36.310 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=828.108ยตs] [2024/04/26 19:21:36.311 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:21:36.311 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:21:36.316 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%2CONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:21:36.319 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/04/26 19:21:36.319 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/04/26 19:21:36.319 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/04/26 19:21:36.319 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/04/26 19:21:36.319 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/04/26 19:21:36.320 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/04/26 19:21:36.322 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:21:36.322 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:21:36.326 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CALLOW_INVALID_DATES%2CIGNORE_SPACE%2CNO_AUTO_VALUE_ON_ZERO%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:21:36.328 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/04/26 19:21:36.328 +08:00] [ERROR] [redo.go:268] ["ignore unsupported DDL"] [ddl="{\"StartTs\":449349017328156695,\"CommitTs\":449349017341001734,\"Query\":\"ALTER TABLE `consistent_replicate_storage_file_large_value`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_file_large_value`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_storage_file_large_value\",\"tbl-name\":\"t1\",\"tbl-id\":108,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:21:36.444 +08:00] [INFO] [main.go:220] ["1 insert success: 800"] [2024/04/26 19:21:36.450 +08:00] [INFO] [main.go:234] ["1 delete success: 400"] [2024/04/26 19:21:36.503 +08:00] [INFO] [main.go:234] ["0 delete success: 400"] [2024/04/26 19:21:36.505 +08:00] [INFO] [main.go:220] ["0 insert success: 800"] [2024/04/26 19:21:36.529 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=449349023606505477] [2024/04/26 19:21:36.529 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=449349023606505477] wait process cdc.test exit for 2-th time... [2024/04/26 19:21:36.829 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=449349023606505477] [2024/04/26 19:21:36.829 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=449349023606505477] [2024/04/26 19:21:36.930 +08:00] [INFO] [redo.go:219] ["apply redo log finishes"] [appliedLogCount=8] [appliedDDLCount=0] [currentCheckpoint=449349023606505477] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/04/26 19:21:36.930 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] Apply redo log successfully cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:21:37 CST 2024] <<<<<< run test case event_filter success! >>>>>> [2024/04/26 19:21:36.931 +08:00] [INFO] [main.go:220] ["1 insert success: 900"] [2024/04/26 19:21:36.963 +08:00] [INFO] [main.go:220] ["0 insert success: 900"] check diff successfully [Fri Apr 26 19:21:37 CST 2024] <<<<<< run test case consistent_replicate_storage_file_large_value success! >>>>>> count(*) 0 table force_replicate_table.t1 exists table force_replicate_table.t2 not exists for 1-th check, retry later table force_replicate_table.t2 exists table force_replicate_table.t3 not exists for 1-th check, retry later table force_replicate_table.t3 exists table force_replicate_table.t4 not exists for 1-th check, retry later table force_replicate_table.t4 exists table force_replicate_table.t5 not exists for 1-th check, retry later [2024/04/26 19:21:42.651 +08:00] [INFO] [main.go:86] ["running ddl test: 5 addDropIndexDDL"] [2024/04/26 19:21:43.216 +08:00] [INFO] [main.go:220] ["0 insert success: 100"] [2024/04/26 19:21:43.216 +08:00] [INFO] [main.go:220] ["1 insert success: 100"] ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2692 0 --:--:-- --:--:-- --:--:-- 2728 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:21:31.345","puller_resolved_ts":"2024-04-26 19:21:24.344","last_synced_ts":"2024-04-26 19:19:14.794","now_ts":"2024-04-26 19:21:31.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:21:31.345","puller_resolved_ts":"2024-04-26' '19:21:24.344","last_synced_ts":"2024-04-26' '19:19:14.794","now_ts":"2024-04-26' '19:21:31.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 15035 7.5 0.0 13847312 141416 ? Sl 19:18 0:11 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 15086 5.1 0.0 13387984 135692 ? Sl 19:18 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1714130499.2693975,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0026588c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714130499.269462,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714130499.385283,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0013cdc00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714130499.3853402,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714130499.4026253,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0023eea80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714130499.4026768,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} [2024/04/26 19:21:43.852 +08:00] [INFO] [main.go:220] ["1 insert success: 200"] [2024/04/26 19:21:43.861 +08:00] [INFO] [main.go:234] ["1 delete success: 100"] [2024/04/26 19:21:43.940 +08:00] [INFO] [main.go:220] ["0 insert success: 200"] [2024/04/26 19:21:43.945 +08:00] [INFO] [main.go:234] ["0 delete success: 100"] {"level":"warn","ts":"2024-04-26T19:21:44.215631+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e2cfc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:21:44.216871+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00002dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:21:44.315462+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00104a1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [2024/04/26 19:21:44.619 +08:00] [INFO] [main.go:220] ["1 insert success: 300"] [2024/04/26 19:21:44.932 +08:00] [INFO] [main.go:220] ["0 insert success: 300"] [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } table force_replicate_table.t5 exists table force_replicate_table.t6 not exists for 1-th check, retry later [2024/04/26 19:21:45.395 +08:00] [INFO] [main.go:220] ["1 insert success: 400"] [2024/04/26 19:21:45.402 +08:00] [INFO] [main.go:234] ["1 delete success: 200"] [2024/04/26 19:21:46.101 +08:00] [INFO] [main.go:220] ["0 insert success: 400"] [2024/04/26 19:21:46.121 +08:00] [INFO] [main.go:234] ["0 delete success: 200"] [2024/04/26 19:21:46.311 +08:00] [INFO] [main.go:220] ["1 insert success: 500"] \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } [2024/04/26 19:21:46.980 +08:00] [INFO] [main.go:220] ["1 insert success: 600"] [2024/04/26 19:21:46.988 +08:00] [INFO] [main.go:234] ["1 delete success: 300"] Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [2024/04/26 19:21:47.071 +08:00] [INFO] [main.go:220] ["0 insert success: 500"] [Pipeline] } =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_replicate_storage_s3/run.sh using Sink-Type: mysql... <<================= * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:24927; Connection refused * Closing connection 0 [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } You are running an older version of MinIO released 3 years ago Update: Run `mc admin update` Attempting encryption of all config, IAM users and policies on MinIO backend [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } table force_replicate_table.t6 exists check_data_subset force_replicate_table.t0 127.0.0.1 4000 127.0.0.1 3306 [Pipeline] // withEnv [Pipeline] } run task successfully check_data_subset force_replicate_table.t1 127.0.0.1 4000 127.0.0.1 3306 Endpoint: http://127.0.0.1:24927 Object API (Amazon S3 compatible): Go: https://docs.min.io/docs/golang-client-quickstart-guide Java: https://docs.min.io/docs/java-client-quickstart-guide Python: https://docs.min.io/docs/python-client-quickstart-guide JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide [Pipeline] // stage run task successfully check_data_subset force_replicate_table.t2 127.0.0.1 4000 127.0.0.1 3306 * About to connect() to 127.0.0.1 port 24927 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 24927 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:24927 > Accept: */* > < HTTP/1.1 403 Forbidden < Accept-Ranges: bytes < Content-Length: 226 < Content-Security-Policy: block-all-mixed-content < Content-Type: application/xml < Server: MinIO/RELEASE.2020-07-27T18-37-02Z < Vary: Origin < X-Amz-Request-Id: 17C9D09DAA9AD731 < X-Xss-Protection: 1; mode=block < Date: Fri, 26 Apr 2024 11:21:48 GMT < { [data not shown] * Connection #0 to host 127.0.0.1 left intact [Pipeline] } Bucket 's3://logbucket/' created The 1 times to try to start tidb cluster... run task successfully check_data_subset force_replicate_table.t3 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t4 127.0.0.1 4000 127.0.0.1 3306 {"level":"warn","ts":"2024-04-26T19:21:50.216286+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e2cfc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:21:50.218995+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00002dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:21:50.316756+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00104a1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} run task successfully check_data_subset force_replicate_table.t5 127.0.0.1 4000 127.0.0.1 3306 run task successfully check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 id=19,a=NULL doesn't exist in downstream table force_replicate_table.t6 run task failed 1-th time, retry later ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed start tidb cluster in /tmp/tidb_cdc_test/consistent_replicate_storage_s3 Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... check_data_subset force_replicate_table.t6 127.0.0.1 4000 127.0.0.1 3306 run task successfully wait process cdc.test exit for 1-th time... [2024/04/26 19:21:54.915 +08:00] [INFO] [main.go:78] ["runDDLTest take 1m3.50003058s"] table mark.finish_mark_0 exists wait process cdc.test exit for 2-th time... table mark.finish_mark_1 not exists for 1-th check, retry later cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:21:56 CST 2024] <<<<<< run test case force_replicate_table success! >>>>>> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:21:56.21813+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e2cfc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:21:56.219444+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00002dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:21:56.318232+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00104a1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table mark.finish_mark_1 not exists for 2-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_1 not exists for 3-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_1 exists table mark.finish_mark_2 not exists for 1-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:22:02.21958+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e2cfc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:22:02.220524+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00002dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:22:02.319817+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00104a1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_2 not exists for 2-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:22:04.206874+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00002dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:22:04.206918+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:22:04.208178+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e2cfc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:22:04.208228+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:22:04.308308+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00104a1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:22:04.308347+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table mark.finish_mark_2 not exists for 3-th check, retry later \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68906dc80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:15161, start at 2024-04-26 19:22:05.083303299 +0800 CST m=+5.916575530 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:05.094 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:05.092 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:05.092 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68906dc80013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:15161, start at 2024-04-26 19:22:05.083303299 +0800 CST m=+5.916575530 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:05.094 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:05.092 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:05.092 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68906f6c000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:15213, start at 2024-04-26 19:22:05.170590688 +0800 CST m=+5.907871227 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:05.180 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:05.147 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:05.147 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_replicate_storage_s3/tiflash/log/proxy.log"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table mark.finish_mark_2 not exists for 4-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:22:08.221387+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e2cfc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:22:08.222418+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00002dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:22:08.321116+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00104a1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table mark.finish_mark_2 not exists for 5-th check, retry later [Fri Apr 26 19:22:09 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_s3 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_s3.1643816440.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data --cluster-id default + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table mark.finish_mark_2 not exists for 6-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:22:12 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea4c8f64-19a6-4491-aca7-dc4c6e271ea3 {"id":"ea4c8f64-19a6-4491-aca7-dc4c6e271ea3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23ec8cf3 ea4c8f64-19a6-4491-aca7-dc4c6e271ea3 /tidb/cdc/default/default/upstream/7362134511393833323 {"id":7362134511393833323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea4c8f64-19a6-4491-aca7-dc4c6e271ea3 {"id":"ea4c8f64-19a6-4491-aca7-dc4c6e271ea3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23ec8cf3 ea4c8f64-19a6-4491-aca7-dc4c6e271ea3 /tidb/cdc/default/default/upstream/7362134511393833323 {"id":7362134511393833323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/ea4c8f64-19a6-4491-aca7-dc4c6e271ea3 {"id":"ea4c8f64-19a6-4491-aca7-dc4c6e271ea3","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23ec8cf3 ea4c8f64-19a6-4491-aca7-dc4c6e271ea3 /tidb/cdc/default/default/upstream/7362134511393833323 {"id":7362134511393833323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table mark.finish_mark_2 not exists for 7-th check, retry later ***************** properties ***************** "mysql.port"="4000" "threadcount"="10" "updateproportion"="0" "mysql.user"="root" "readallfields"="true" "requestdistribution"="uniform" "workload"="core" "recordcount"="5000" "dotransactions"="false" "readproportion"="0" "fieldcount"="100" "insertproportion"="0" "mysql.db"="consistent_replicate_storage_s3" "scanproportion"="0" "mysql.host"="127.0.0.1" "operationcount"="0" ********************************************** 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:22:14.222906+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00002dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:22:14.223064+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e2cfc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1714130534.2707062,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0026588c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714130534.2707467,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:22:14.322938+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00104a1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1714130534.3869379,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0013cdc00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714130534.3869667,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714130534.404021,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0023eea80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714130534.4040623,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} table mark.finish_mark_2 exists table mark.finish_mark_3 not exists for 1-th check, retry later Run finished, takes 3.393699367s INSERT - Takes(s): 3.4, Count: 5000, OPS: 1478.7, Avg(us): 6242, Min(us): 1736, Max(us): 205748, 95th(us): 42000, 99th(us): 60000 table mark.finish_mark_3 not exists for 2-th check, retry later table mark.finish_mark_3 not exists for 3-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:22:20.224112+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000e2cfc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:22:20.224627+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00002dc00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:22:20.323854+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00104a1c0/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} table consistent_replicate_storage_s3.usertable exists table consistent_replicate_storage_s3.check1 not exists for 1-th check, retry later table mark.finish_mark_3 not exists for 4-th check, retry later 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' ++ jq -r .error_code + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test table consistent_replicate_storage_s3.check1 not exists for 2-th check, retry later wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster table mark.finish_mark_3 not exists for 5-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 3-th check, retry later table mark.finish_mark_3 not exists for 6-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 4-th check, retry later table mark.finish_mark_3 exists table mark.finish_mark_4 not exists for 1-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 5-th check, retry later table mark.finish_mark_4 not exists for 2-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 6-th check, retry later + run_case_with_unavailable_tikv conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory table mark.finish_mark_4 not exists for 3-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 7-th check, retry later table mark.finish_mark_4 not exists for 4-th check, retry later chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... table consistent_replicate_storage_s3.check1 not exists for 8-th check, retry later Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... table consistent_replicate_storage_s3.check1 not exists for 9-th check, retry later table mark.finish_mark_4 not exists for 5-th check, retry later table consistent_replicate_storage_s3.check1 not exists for 10-th check, retry later table mark.finish_mark_4 not exists for 6-th check, retry later Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release table consistent_replicate_storage_s3.check1 not exists for 11-th check, retry later table mark.finish_mark_4 not exists for 7-th check, retry later Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.check1 not exists for 12-th check, retry later table mark.finish_mark_4 not exists for 8-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.check1 not exists for 13-th check, retry later table mark.finish_mark_4 exists table mark.finish_mark not exists for 1-th check, retry later ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) table consistent_replicate_storage_s3.check1 not exists for 14-th check, retry later table mark.finish_mark not exists for 2-th check, retry later VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6892f3e00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:18279, start at 2024-04-26 19:22:46.397333749 +0800 CST m=+5.015735732 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:46.404 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:46.392 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:46.392 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6892f3e00004 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:18279, start at 2024-04-26 19:22:46.397333749 +0800 CST m=+5.015735732 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:46.404 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:46.392 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:46.392 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6892f55c0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:18357, start at 2024-04-26 19:22:46.525161057 +0800 CST m=+5.093727295 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:24:46.532 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:22:46.487 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:12:46.487 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } table consistent_replicate_storage_s3.check1 not exists for 15-th check, retry later table mark.finish_mark not exists for 3-th check, retry later + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19711.out cli tso query --pd=http://127.0.0.1:2379 table consistent_replicate_storage_s3.check1 not exists for 16-th check, retry later table mark.finish_mark not exists for 4-th check, retry later + set +x + tso='449349044139982849 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349044139982849 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349044139982849 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:22:51 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.1974719749.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 table consistent_replicate_storage_s3.check1 exists table consistent_replicate_storage_s3.t2 not exists for 1-th check, retry later table mark.finish_mark not exists for 5-th check, retry later + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:22:54 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2 {"id":"d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a249196f3 d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2 /tidb/cdc/default/default/upstream/7362134688269639106 {"id":7362134688269639106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2 {"id":"d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a249196f3 d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2 /tidb/cdc/default/default/upstream/7362134688269639106 {"id":7362134688269639106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2 {"id":"d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a249196f3 d0ab707c-ae32-49b1-b888-3ad3ce1f6fa2 /tidb/cdc/default/default/upstream/7362134688269639106 {"id":7362134688269639106,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349044139982849 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.19794.out cli changefeed create --start-ts=449349044139982849 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml table consistent_replicate_storage_s3.t2 not exists for 2-th check, retry later Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362134688269639106,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:22:54.892940847+08:00","start_ts":449349044139982849,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349044139982849,"checkpoint_ts":449349044139982849,"checkpoint_time":"2024-04-26 19:22:49.992"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... table mark.finish_mark not exists for 6-th check, retry later + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 exists + sleep 5 table consistent_replicate_storage_s3.t2 not exists for 3-th check, retry later table mark.finish_mark not exists for 7-th check, retry later table consistent_replicate_storage_s3.t2 not exists for 4-th check, retry later table mark.finish_mark exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... table consistent_replicate_storage_s3.t2 not exists for 5-th check, retry later wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:23:01 CST 2024] <<<<<< run test case multi_source success! >>>>>> + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 17775 18.0 0.4 3762652 1609448 ? Sl 19:22 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 17776 17.9 0.4 3773912 1614248 ? Sl 19:22 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 17777 24.9 0.4 3812316 1683060 ? Sl 19:22 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 17779 24.3 0.4 3807708 1675372 ? Sl 19:22 0:05 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2215 0 --:--:-- --:--:-- --:--:-- 2229 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:23:01.443","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:22:56.793","now_ts":"2024-04-26 19:23:01.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:23:01.443","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:22:56.793","now_ts":"2024-04-26' '19:23:01.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:23:01.443","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:22:56.793","now_ts":"2024-04-26' '19:23:01.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 table consistent_replicate_storage_s3.t2 not exists for 6-th check, retry later table consistent_replicate_storage_s3.t2 exists check diff successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [Fri Apr 26 19:23:07 CST 2024] <<<<<< START cdc server in consistent_replicate_storage_s3 case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true)' + '[' -z '' ']' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_replicate_storage_s3.1679716799.out server --log-file /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data --cluster-id default + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:23:10 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/b1bdf734-875e-44d3-b46a-631ecc8930bb {UpstreamID:7362134511393833323 Namespace:default ID:b1bdf734-875e-44d3-b46a-631ecc8930bb SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:22:12.967169577 +0800 CST StartTs:449349034401333252 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00390ecf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349034427547652} {CheckpointTs:449349049225576450 MinTableBarrierTs:449349049238683652 AdminJobType:noop} span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349049238683653, checkpointTs: 449349049225576450, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fc28d891-7c18-4cd1-8a6a-3e8512054c48 {"id":"fc28d891-7c18-4cd1-8a6a-3e8512054c48","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23ec8e19 fc28d891-7c18-4cd1-8a6a-3e8512054c48 /tidb/cdc/default/default/changefeed/info/b1bdf734-875e-44d3-b46a-631ecc8930bb {"upstream-id":7362134511393833323,"namespace":"default","changefeed-id":"b1bdf734-875e-44d3-b46a-631ecc8930bb","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:22:12.967169577+08:00","start-ts":449349034401333252,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349034427547652} /tidb/cdc/default/default/changefeed/status/b1bdf734-875e-44d3-b46a-631ecc8930bb {"checkpoint-ts":449349049225576450,"min-table-barrier-ts":449349049238683652,"admin-job-type":0} /tidb/cdc/default/default/task/position/fc28d891-7c18-4cd1-8a6a-3e8512054c48/b1bdf734-875e-44d3-b46a-631ecc8930bb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362134511393833323 {"id":7362134511393833323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/b1bdf734-875e-44d3-b46a-631ecc8930bb {UpstreamID:7362134511393833323 Namespace:default ID:b1bdf734-875e-44d3-b46a-631ecc8930bb SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:22:12.967169577 +0800 CST StartTs:449349034401333252 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00390ecf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349034427547652} {CheckpointTs:449349049225576450 MinTableBarrierTs:449349049238683652 AdminJobType:noop} span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349049238683653, checkpointTs: 449349049225576450, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fc28d891-7c18-4cd1-8a6a-3e8512054c48 {"id":"fc28d891-7c18-4cd1-8a6a-3e8512054c48","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23ec8e19 fc28d891-7c18-4cd1-8a6a-3e8512054c48 /tidb/cdc/default/default/changefeed/info/b1bdf734-875e-44d3-b46a-631ecc8930bb {"upstream-id":7362134511393833323,"namespace":"default","changefeed-id":"b1bdf734-875e-44d3-b46a-631ecc8930bb","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:22:12.967169577+08:00","start-ts":449349034401333252,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349034427547652} /tidb/cdc/default/default/changefeed/status/b1bdf734-875e-44d3-b46a-631ecc8930bb {"checkpoint-ts":449349049225576450,"min-table-barrier-ts":449349049238683652,"admin-job-type":0} /tidb/cdc/default/default/task/position/fc28d891-7c18-4cd1-8a6a-3e8512054c48/b1bdf734-875e-44d3-b46a-631ecc8930bb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362134511393833323 {"id":7362134511393833323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/b1bdf734-875e-44d3-b46a-631ecc8930bb {UpstreamID:7362134511393833323 Namespace:default ID:b1bdf734-875e-44d3-b46a-631ecc8930bb SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:22:12.967169577 +0800 CST StartTs:449349034401333252 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc00390ecf0 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349034427547652} {CheckpointTs:449349049225576450 MinTableBarrierTs:449349049238683652 AdminJobType:noop} span: {table_id:111,start_key:7480000000000000ff6f5f720000000000fa,end_key:7480000000000000ff6f5f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:110,start_key:7480000000000000ff6e5f720000000000fa,end_key:7480000000000000ff6e5f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating span: {table_id:104,start_key:7480000000000000ff685f720000000000fa,end_key:7480000000000000ff685f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349049238683653, checkpointTs: 449349049225576450, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349049225576450, checkpointTs: 449349049225576450, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349049238683652, checkpointTs: 449349049225576450, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/fc28d891-7c18-4cd1-8a6a-3e8512054c48 {"id":"fc28d891-7c18-4cd1-8a6a-3e8512054c48","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a23ec8e19 fc28d891-7c18-4cd1-8a6a-3e8512054c48 /tidb/cdc/default/default/changefeed/info/b1bdf734-875e-44d3-b46a-631ecc8930bb {"upstream-id":7362134511393833323,"namespace":"default","changefeed-id":"b1bdf734-875e-44d3-b46a-631ecc8930bb","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:22:12.967169577+08:00","start-ts":449349034401333252,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/","use-file-backend":false,"compression":"","flush-concurrency":2,"memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349034427547652} /tidb/cdc/default/default/changefeed/status/b1bdf734-875e-44d3-b46a-631ecc8930bb {"checkpoint-ts":449349049225576450,"min-table-barrier-ts":449349049238683652,"admin-job-type":0} /tidb/cdc/default/default/task/position/fc28d891-7c18-4cd1-8a6a-3e8512054c48/b1bdf734-875e-44d3-b46a-631ecc8930bb {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362134511393833323 {"id":7362134511393833323,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } check_redo_resolved_ts b1bdf734-875e-44d3-b46a-631ecc8930bb 449349055346376706 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/b1bdf734-875e-44d3-b46a-631ecc8930bb/meta global resolved ts 449349055005851651 not forward to 449349055346376706 run task failed 1-th time, retry later check_redo_resolved_ts b1bdf734-875e-44d3-b46a-631ecc8930bb 449349055346376706 s3://logbucket/test-changefeed?endpoint=http://127.0.0.1:24927/ /tmp/tidb_cdc_test/consistent_replicate_storage_s3/cdc_data/redo/b1bdf734-875e-44d3-b46a-631ecc8930bb/meta run task successfully wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... wait process cdc.test exit for 3-th time... cdc.test: no process found wait process cdc.test exit for 4-th time... process cdc.test already exit [2024/04/26 19:23:37.090 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/04/26 19:23:37.091 +08:00] [INFO] [s3.go:407] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/04/26 19:23:37.097 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/04/26 19:23:37.097 +08:00] [INFO] [redo.go:163] ["apply redo log starts"] [checkpointTs=449349049710542863] [resolvedTs=449349055530139652] [2024/04/26 19:23:37.098 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:23:37.098 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:23:37.099 +08:00] [INFO] [s3.go:407] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/04/26 19:23:37.099 +08:00] [INFO] [s3.go:407] ["succeed to get bucket region from s3"] ["bucket region"=us-east-1] [2024/04/26 19:23:37.104 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%2CIGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:23:37.105 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=7.203972ms] [2024/04/26 19:23:37.112 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/04/26 19:23:37.112 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/04/26 19:23:37.113 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:23:37.113 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:23:37.119 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22IGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%2CONLY_FULL_GROUP_BY%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:23:37.125 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/04/26 19:23:37.839 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=741.64128ms] [2024/04/26 19:23:37.840 +08:00] [ERROR] [redo.go:268] ["ignore unsupported DDL"] [ddl="{\"StartTs\":449349049710542857,\"CommitTs\":449349049710542863,\"Query\":\"ALTER TABLE `consistent_replicate_storage_s3`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `consistent_replicate_storage_s3`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"consistent_replicate_storage_s3\",\"tbl-name\":\"t1\",\"tbl-id\":108,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:23:38.685 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=449349055530139652] [2024/04/26 19:23:38.685 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=449349055530139652] [2024/04/26 19:23:38.786 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=449349055530139652] [2024/04/26 19:23:38.786 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=449349055530139652] [2024/04/26 19:23:39.488 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:118,start_key:7480000000000000ff765f720000000000fa,end_key:7480000000000000ff765f730000000000fa}] [checkpointTs=449349055530139652] [2024/04/26 19:23:39.489 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:118,start_key:7480000000000000ff765f720000000000fa,end_key:7480000000000000ff765f730000000000fa}] [checkpointTs=449349055530139652] [2024/04/26 19:23:39.589 +08:00] [INFO] [redo.go:219] ["apply redo log finishes"] [appliedLogCount=5008] [appliedDDLCount=0] [currentCheckpoint=449349055530139652] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/04/26 19:23:39.589 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] Apply redo log successfully check diff successfully [Fri Apr 26 19:23:40 CST 2024] <<<<<< run test case consistent_replicate_storage_s3 success! >>>>>> count(*) 5000 Exiting on signal: INTERRUPT =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/consistent_partition_table/run.sh using Sink-Type: mysql... <<================= The 1 times to try to start tidb cluster... start tidb cluster in /tmp/tidb_cdc_test/consistent_partition_table Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6897aa880012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:17937, start at 2024-04-26 19:24:03.648961789 +0800 CST m=+5.084462668 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:26:03.656 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:24:03.618 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:14:03.618 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6897aa880012 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:17937, start at 2024-04-26 19:24:03.648961789 +0800 CST m=+5.084462668 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:26:03.656 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:24:03.618 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:14:03.618 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c6897ae2c0005 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-0r83d, pid:18008, start at 2024-04-26 19:24:03.857031907 +0800 CST m=+5.200180631 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:26:03.864 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:24:03.851 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:14:03.851 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/consistent_partition_table/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } [Fri Apr 26 19:24:06 CST 2024] <<<<<< START cdc server in consistent_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_partition_table.1931219314.out server --log-file /tmp/tidb_cdc_test/consistent_partition_table/cdcpartition_table.server1.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_partition_table/cdc_datapartition_table.server1 --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:24:10 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b0ce6e22-d818-4e23-8baa-a35b50a86833 {"id":"b0ce6e22-d818-4e23-8baa-a35b50a86833","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a25c2f4f1 b0ce6e22-d818-4e23-8baa-a35b50a86833 /tidb/cdc/default/default/upstream/7362135021683728419 {"id":7362135021683728419,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b0ce6e22-d818-4e23-8baa-a35b50a86833 {"id":"b0ce6e22-d818-4e23-8baa-a35b50a86833","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a25c2f4f1 b0ce6e22-d818-4e23-8baa-a35b50a86833 /tidb/cdc/default/default/upstream/7362135021683728419 {"id":7362135021683728419,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b0ce6e22-d818-4e23-8baa-a35b50a86833 {"id":"b0ce6e22-d818-4e23-8baa-a35b50a86833","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a25c2f4f1 b0ce6e22-d818-4e23-8baa-a35b50a86833 /tidb/cdc/default/default/upstream/7362135021683728419 {"id":7362135021683728419,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x table partition_table2.t2 not exists for 1-th check, retry later table partition_table2.t2 not exists for 2-th check, retry later table partition_table2.t2 not exists for 3-th check, retry later table partition_table2.t2 not exists for 4-th check, retry later table partition_table2.t2 not exists for 5-th check, retry later table partition_table2.t2 not exists for 6-th check, retry later table partition_table2.t2 not exists for 7-th check, retry later table partition_table2.t2 exists wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit [Fri Apr 26 19:24:34 CST 2024] <<<<<< START cdc server in consistent_partition_table case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql/MySQLSinkHangLongTime=return(true);github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql/MySQLSinkExecDDLDelay=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.consistent_partition_table.1949919501.out server --log-file /tmp/tidb_cdc_test/consistent_partition_table/cdcpartition_table.server2.log --log-level debug --data-dir /tmp/tidb_cdc_test/consistent_partition_table/cdc_datapartition_table.server2 --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:24:37 GMT < Content-Type: text/plain; charset=utf-8 < Transfer-Encoding: chunked < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: changefeedID: default/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {UpstreamID:7362135021683728419 Namespace:default ID:34d3b2d8-8ae0-4920-b19b-8f785e18b79c SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:24:10.222508248 +0800 CST StartTs:449349065131687941 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0036eec60 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349065170747393} {CheckpointTs:449349070531854344 MinTableBarrierTs:449349072104718344 AdminJobType:noop} span: {table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c6b41eff-27fc-4273-be65-700f4d005206 {"id":"c6b41eff-27fc-4273-be65-700f4d005206","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a25c2f5cb c6b41eff-27fc-4273-be65-700f4d005206 /tidb/cdc/default/default/changefeed/info/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"upstream-id":7362135021683728419,"namespace":"default","changefeed-id":"34d3b2d8-8ae0-4920-b19b-8f785e18b79c","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:24:10.222508248+08:00","start-ts":449349065131687941,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349065170747393} /tidb/cdc/default/default/changefeed/status/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"checkpoint-ts":449349070531854344,"min-table-barrier-ts":449349072104718344,"admin-job-type":0} /tidb/cdc/default/default/task/position/c6b41eff-27fc-4273-be65-700f4d005206/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362135021683728419 {"id":7362135021683728419,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {UpstreamID:7362135021683728419 Namespace:default ID:34d3b2d8-8ae0-4920-b19b-8f785e18b79c SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:24:10.222508248 +0800 CST StartTs:449349065131687941 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0036eec60 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349065170747393} {CheckpointTs:449349070531854344 MinTableBarrierTs:449349072104718344 AdminJobType:noop} span: {table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c6b41eff-27fc-4273-be65-700f4d005206 {"id":"c6b41eff-27fc-4273-be65-700f4d005206","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a25c2f5cb c6b41eff-27fc-4273-be65-700f4d005206 /tidb/cdc/default/default/changefeed/info/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"upstream-id":7362135021683728419,"namespace":"default","changefeed-id":"34d3b2d8-8ae0-4920-b19b-8f785e18b79c","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:24:10.222508248+08:00","start-ts":449349065131687941,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349065170747393} /tidb/cdc/default/default/changefeed/status/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"checkpoint-ts":449349070531854344,"min-table-barrier-ts":449349072104718344,"admin-job-type":0} /tidb/cdc/default/default/task/position/c6b41eff-27fc-4273-be65-700f4d005206/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362135021683728419 {"id":7362135021683728419,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + echo ' *** owner info ***: *** processors info ***: changefeedID: default/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {UpstreamID:7362135021683728419 Namespace:default ID:34d3b2d8-8ae0-4920-b19b-8f785e18b79c SinkURI:mysql://normal:123456@127.0.0.1:3306/ CreateTime:2024-04-26 19:24:10.222508248 +0800 CST StartTs:449349065131687941 TargetTs:0 AdminJobType:noop Engine:unified SortDir: Config:0xc0036eec60 State:normal Error: Warning: CreatorVersion:v7.5.1-57-g10f68e69e Epoch:449349065170747393} {CheckpointTs:449349070531854344 MinTableBarrierTs:449349072104718344 AdminJobType:noop} span: {table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating span: {table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}, resolvedTs: 449349070531854344, checkpointTs: 449349070531854344, state: Replicating *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/c6b41eff-27fc-4273-be65-700f4d005206 {"id":"c6b41eff-27fc-4273-be65-700f4d005206","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a25c2f5cb c6b41eff-27fc-4273-be65-700f4d005206 /tidb/cdc/default/default/changefeed/info/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"upstream-id":7362135021683728419,"namespace":"default","changefeed-id":"34d3b2d8-8ae0-4920-b19b-8f785e18b79c","sink-uri":"mysql://normal:123456@127.0.0.1:3306/","create-time":"2024-04-26T19:24:10.222508248+08:00","start-ts":449349065131687941,"target-ts":0,"admin-job-type":0,"sort-engine":"","sort-dir":"","config":{"memory-quota":1073741824,"case-sensitive":false,"force-replicate":false,"check-gc-safe-point":true,"enable-sync-point":false,"ignore-ineligible-table":false,"bdr-mode":false,"sync-point-interval":600000000000,"sync-point-retention":86400000000000,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"event-filters":null},"mounter":{"worker-num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include-commit-ts":false,"binary-encoding-method":"base64"},"encoder-concurrency":32,"terminator":"\r\n","date-separator":"day","enable-partition-separator":true,"enable-kafka-sink-v2":false,"only-output-updated-columns":false,"delete-only-output-handle-key-columns":false,"advance-timeout-in-sec":150},"consistent":{"level":"eventual","max-log-size":64,"flush-interval":2000,"meta-flush-interval":200,"encoding-worker-num":16,"flush-worker-num":8,"storage":"file:///tmp/tidb_cdc_test/consistent_partition_table/redo","use-file-backend":false,"compression":"","memory-usage":{"memory-quota-percentage":50,"event-cache-percentage":0}},"scheduler":{"enable-table-across-nodes":false,"region-threshold":100000,"write-key-threshold":0,"region-per-span":0},"integrity":{"integrity-check-level":"none","corruption-handle-level":"warn"},"changefeed-error-stuck-duration":1800000000000,"sql-mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced-status":{"synced-check-interval":300,"checkpoint-interval":15}},"state":"normal","error":null,"warning":null,"creator-version":"v7.5.1-57-g10f68e69e","epoch":449349065170747393} /tidb/cdc/default/default/changefeed/status/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"checkpoint-ts":449349070531854344,"min-table-barrier-ts":449349072104718344,"admin-job-type":0} /tidb/cdc/default/default/task/position/c6b41eff-27fc-4273-be65-700f4d005206/34d3b2d8-8ae0-4920-b19b-8f785e18b79c {"checkpoint-ts":0,"resolved-ts":0,"count":0,"error":null,"warning":null} /tidb/cdc/default/default/upstream/7362135021683728419 {"id":7362135021683728419,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + break + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8849 0 --:--:-- --:--:-- --:--:-- 8925 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:23:01.443","puller_resolved_ts":"2024-04-26 19:23:01.443","last_synced_ts":"2024-04-26 19:22:56.793","now_ts":"2024-04-26 19:25:12.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:23:01.443","puller_resolved_ts":"2024-04-26' '19:23:01.443","last_synced_ts":"2024-04-26' '19:22:56.793","now_ts":"2024-04-26' '19:25:12.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:23:01.443","puller_resolved_ts":"2024-04-26' '19:23:01.443","last_synced_ts":"2024-04-26' '19:22:56.793","now_ts":"2024-04-26' '19:25:12.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c689d46800013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:20857, start at 2024-04-26 19:25:35.561366112 +0800 CST m=+5.122647042 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:27:35.568 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:25:35.569 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:15:35.569 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c689d46800013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:20857, start at 2024-04-26 19:25:35.561366112 +0800 CST m=+5.122647042 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:27:35.568 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:25:35.569 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:15:35.569 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c689d4804000f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:20944, start at 2024-04-26 19:25:35.63109253 +0800 CST m=+5.121999395 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:27:35.637 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:25:35.617 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:15:35.617 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22266.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349088449658881 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349088449658881 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349088449658881 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:25:40 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + (( i = 0 )) + GO_FAILPOINTS= + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2230422306.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:25:43 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2829d49b-cf92-445a-9153-b147b8bc63b7 {"id":"2829d49b-cf92-445a-9153-b147b8bc63b7","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2725a8f4 2829d49b-cf92-445a-9153-b147b8bc63b7 /tidb/cdc/default/default/upstream/7362135413101423216 {"id":7362135413101423216,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2829d49b-cf92-445a-9153-b147b8bc63b7 {"id":"2829d49b-cf92-445a-9153-b147b8bc63b7","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2725a8f4 2829d49b-cf92-445a-9153-b147b8bc63b7 /tidb/cdc/default/default/upstream/7362135413101423216 {"id":7362135413101423216,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/2829d49b-cf92-445a-9153-b147b8bc63b7 {"id":"2829d49b-cf92-445a-9153-b147b8bc63b7","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2725a8f4 2829d49b-cf92-445a-9153-b147b8bc63b7 /tidb/cdc/default/default/upstream/7362135413101423216 {"id":7362135413101423216,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349088449658881 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.22353.out cli changefeed create --start-ts=449349088449658881 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362135413101423216,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:25:43.925833589+08:00","start_ts":449349088449658881,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349088449658881,"checkpoint_ts":449349088449658881,"checkpoint_time":"2024-04-26 19:25:39.020"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 20857 10.8 0.0 2665460 216372 ? Sl 19:25 0:02 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714130730432614226.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 20861 3.5 0.0 2591312 162388 ? Sl 19:25 0:00 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714130730435512697.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 20944 11.5 0.0 2599924 209100 ? Sl 19:25 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714130730502904244.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2584 0 --:--:-- --:--:-- --:--:-- 2612 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:25:51.920","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:25:45.369","now_ts":"2024-04-26 19:25:52.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:25:51.920","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:25:45.369","now_ts":"2024-04-26' '19:25:52.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:25:51.920","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:25:45.369","now_ts":"2024-04-26' '19:25:52.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit # diff Configuration. check-thread-count = 4 export-fix-sql = true check-struct-only = false [task] output-dir = "/tmp/tidb_cdc_test/partition_table/sync_diff/output" source-instances = ["mysql1"] target-instance = "tidb0" target-check-tables = ["partition_table.?*"] [data-sources] [data-sources.mysql1] host = "127.0.0.1" port = 4000 user = "root" password = "" [data-sources.tidb0] host = "127.0.0.1" port = 3306 user = "root" password = "" [2024/04/26 19:26:42.790 +08:00] [INFO] [helper.go:54] ["init log"] [file=] [level=info] [2024/04/26 19:26:42.790 +08:00] [INFO] [mem_quota.go:82] ["New memory quota"] [namespace=default] [changefeed=redo-applier] [total=1073741824] [2024/04/26 19:26:42.791 +08:00] [INFO] [redo.go:163] ["apply redo log starts"] [checkpointTs=449349070531854344] [resolvedTs=449349104374382601] [2024/04/26 19:26:42.791 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:26:42.792 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:26:42.792 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=row] [duration=1.015325ms] [2024/04/26 19:26:42.792 +08:00] [INFO] [file.go:114] ["succeed to download and sort redo logs"] [type=ddl] [duration=1.025299ms] [2024/04/26 19:26:42.796 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CIGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:26:42.799 +08:00] [INFO] [mysql.go:198] ["MySQL backends is created"] [changefeed=.] [workerCount=16] [forceReplicate=false] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=0] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=3] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=2] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=1] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=8] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=9] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=6] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=7] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=11] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=13] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=10] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=12] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=15] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=14] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=5] [2024/04/26 19:26:42.799 +08:00] [INFO] [worker.go:105] ["Transaction dmlSink worker starts"] [changefeedID=.] [workerID=4] [2024/04/26 19:26:42.800 +08:00] [INFO] [tz.go:35] ["Use the timezone of the TiCDC server machine"] [timezoneName=System] [timezone=Asia/Shanghai] [2024/04/26 19:26:42.800 +08:00] [WARN] [config.go:380] ["Because time-zone is not specified, the timezone of the TiCDC server will be used. We recommend that you specify the time-zone explicitly. Please make sure that the timezone of the TiCDC server, sink-uri and the downstream database are consistent. If the downstream database does not load the timezone information, you can refer to https://dev.mysql.com/doc/refman/8.0/en/mysql-tzinfo-to-sql.html."] [timezone=Asia/Shanghai] [2024/04/26 19:26:42.806 +08:00] [INFO] [db_helper.go:175] ["sink uri is configured"] [dsn="normal:******@tcp(127.0.0.1:3306)/?interpolateParams=true&multiStatements=true&allow_auto_random_explicit_insert=1&charset=utf8mb4&foreign_key_checks=0&maxAllowedPacket=0&readTimeout=2m&sql_mode=%22ONLY_FULL_GROUP_BY%2CIGNORE_SPACE%2CNO_ENGINE_SUBSTITUTION%2CALLOW_INVALID_DATES%2CNO_AUTO_VALUE_ON_ZERO%22&tidb_enable_external_ts_read=%22OFF%22&tidb_placement_mode=%22ignore%22&tidb_txn_mode=optimistic&time_zone=%22Asia%2FShanghai%22&timeout=2m&transaction_isolation=%22READ-COMMITTED%22&writeTimeout=2m"] [2024/04/26 19:26:42.807 +08:00] [INFO] [mysql_ddl_sink.go:99] ["MySQL DDL sink is created"] [namespace=] [changefeed=] [2024/04/26 19:26:42.808 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449349072366600202,\"CommitTs\":449349072366600211,\"Query\":\"alter table t1 drop partition p1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":20,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:26:44.815 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449349072366600202,\"CommitTs\":449349072366600211,\"Query\":\"alter table t1 drop partition p1\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":20,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:26:45.036 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="alter table t1 drop partition p1"] [duration=221.255769ms] [namespace=] [changefeed=] [2024/04/26 19:26:45.036 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449349072484565000,\"CommitTs\":449349072484565005,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:26:45.236 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449349072484565000,\"CommitTs\":449349072484565005,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:26:45.388 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table`.`t2`"] [duration=151.921476ms] [namespace=] [changefeed=] [2024/04/26 19:26:45.388 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449349072615637009,\"CommitTs\":449349072628744193,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:26:45.790 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449349072615637009,\"CommitTs\":449349072628744193,\"Query\":\"ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":42,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:26:45.936 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE `partition_table`.`t1` EXCHANGE PARTITION `p3` WITH TABLE `partition_table2`.`t2`"] [duration=145.966373ms] [namespace=] [changefeed=] [2024/04/26 19:26:45.936 +08:00] [WARN] [redo.go:283] ["apply DDL"] [ddl="{\"StartTs\":449349073008853002,\"CommitTs\":449349073008853006,\"Query\":\"ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":64,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [2024/04/26 19:26:46.336 +08:00] [INFO] [mysql_ddl_sink.go:186] ["Start exec DDL"] [DDL="{\"StartTs\":449349073008853002,\"CommitTs\":449349073008853006,\"Query\":\"ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))\",\"TableInfo\":{\"SchemaID\":0,\"TableName\":{\"db-name\":\"partition_table\",\"tbl-name\":\"t1\",\"tbl-id\":111,\"is-partition\":true},\"Version\":0,\"RowColumnsOffset\":null,\"ColumnsFlag\":null,\"HandleIndexID\":0,\"IndexColumnsOffset\":null},\"PreTableInfo\":null,\"Type\":64,\"Done\":{},\"Charset\":\"\",\"Collate\":\"\"}"] [namespace=] [changefeed=] [2024/04/26 19:26:47.507 +08:00] [INFO] [mysql_ddl_sink.go:221] ["Exec DDL succeeded"] [sql="ALTER TABLE t1 REORGANIZE PARTITION p0,p2 INTO (PARTITION p0 VALUES LESS THAN (5), PARTITION p1 VALUES LESS THAN (10), PARTITION p2 VALUES LESS THAN (21))"] [duration=1.170973613s] [namespace=] [changefeed=] [2024/04/26 19:26:47.507 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.508 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:109,start_key:7480000000000000ff6d5f720000000000fa,end_key:7480000000000000ff6d5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.608 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.608 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:108,start_key:7480000000000000ff6c5f720000000000fa,end_key:7480000000000000ff6c5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.708 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.708 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:114,start_key:7480000000000000ff725f720000000000fa,end_key:7480000000000000ff725f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.808 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.808 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:116,start_key:7480000000000000ff745f720000000000fa,end_key:7480000000000000ff745f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.909 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:47.909 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:123,start_key:7480000000000000ff7b5f720000000000fa,end_key:7480000000000000ff7b5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.009 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.009 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:107,start_key:7480000000000000ff6b5f720000000000fa,end_key:7480000000000000ff6b5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.111 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.111 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:105,start_key:7480000000000000ff695f720000000000fa,end_key:7480000000000000ff695f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.212 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.212 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:106,start_key:7480000000000000ff6a5f720000000000fa,end_key:7480000000000000ff6a5f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.312 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.312 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:112,start_key:7480000000000000ff705f720000000000fa,end_key:7480000000000000ff705f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.412 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.412 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:113,start_key:7480000000000000ff715f720000000000fa,end_key:7480000000000000ff715f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.512 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.512 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:117,start_key:7480000000000000ff755f720000000000fa,end_key:7480000000000000ff755f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.614 +08:00] [INFO] [table_sink_impl.go:257] ["Table sink stopped"] [namespace=default] [changefeed=redo-applier] [span={table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.614 +08:00] [INFO] [table_sink_impl.go:239] ["Stopping table sink"] [namespace=default] [changefeed=redo-applier] [span={table_id:119,start_key:7480000000000000ff775f720000000000fa,end_key:7480000000000000ff775f730000000000fa}] [checkpointTs=449349104374382601] [2024/04/26 19:26:48.714 +08:00] [INFO] [redo.go:219] ["apply redo log finishes"] [appliedLogCount=45] [appliedDDLCount=4] [currentCheckpoint=449349104374382601] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=4] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=14] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=15] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=3] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=9] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=0] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=5] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=10] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=6] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=12] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=7] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=13] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=11] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=1] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=2] [2024/04/26 19:26:48.714 +08:00] [INFO] [worker.go:120] ["Transaction dmlSink worker exits as canceled"] [changefeedID=.] [workerID=8] Apply redo log successfully check diff successfully [Fri Apr 26 19:26:48 CST 2024] <<<<<< run test case consistent_partition_table success! >>>>>> Database INFORMATION_SCHEMA METRICS_SCHEMA PERFORMANCE_SCHEMA mysql partition_table partition_table2 test =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_big_txn_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:26:58 CST 2024] <<<<<< run test case kafka_big_txn_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/kafka_big_messages_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:27:01 CST 2024] <<<<<< run test case kafka_big_messages_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_tables_ddl_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:27:04 CST 2024] <<<<<< run test case multi_tables_ddl_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/multi_topics_v2/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:27:06 CST 2024] <<<<<< run test case multi_topics_v2 success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup/run.sh using Sink-Type: mysql... <<================= +++ dirname /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup/run.sh ++ cd /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup ++ pwd + CUR=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup + source /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/storage_cleanup/../_utils/test_prepare ++ UP_TIDB_HOST=127.0.0.1 ++ UP_TIDB_PORT=4000 ++ UP_TIDB_OTHER_PORT=4001 ++ UP_TIDB_STATUS=10080 ++ UP_TIDB_OTHER_STATUS=10081 ++ DOWN_TIDB_HOST=127.0.0.1 ++ DOWN_TIDB_PORT=3306 ++ DOWN_TIDB_STATUS=20080 ++ TLS_TIDB_HOST=127.0.0.1 ++ TLS_TIDB_PORT=3307 ++ TLS_TIDB_STATUS=30080 ++ UP_PD_HOST_1=127.0.0.1 ++ UP_PD_PORT_1=2379 ++ UP_PD_PEER_PORT_1=2380 ++ UP_PD_HOST_2=127.0.0.1 ++ UP_PD_PORT_2=2679 ++ UP_PD_PEER_PORT_2=2680 ++ UP_PD_HOST_3=127.0.0.1 ++ UP_PD_PORT_3=2779 ++ UP_PD_PEER_PORT_3=2780 ++ DOWN_PD_HOST=127.0.0.1 ++ DOWN_PD_PORT=2479 ++ DOWN_PD_PEER_PORT=2480 ++ TLS_PD_HOST=127.0.0.1 ++ TLS_PD_PORT=2579 ++ TLS_PD_PEER_PORT=2580 ++ UP_TIKV_HOST_1=127.0.0.1 ++ UP_TIKV_PORT_1=20160 ++ UP_TIKV_STATUS_PORT_1=20181 ++ UP_TIKV_HOST_2=127.0.0.1 ++ UP_TIKV_PORT_2=20161 ++ UP_TIKV_STATUS_PORT_2=20182 ++ UP_TIKV_HOST_3=127.0.0.1 ++ UP_TIKV_PORT_3=20162 ++ UP_TIKV_STATUS_PORT_3=20183 ++ DOWN_TIKV_HOST=127.0.0.1 ++ DOWN_TIKV_PORT=21160 ++ DOWN_TIKV_STATUS_PORT=21180 ++ TLS_TIKV_HOST=127.0.0.1 ++ TLS_TIKV_PORT=22160 ++ TLS_TIKV_STATUS_PORT=22180 +++ cat /tmp/tidb_cdc_test/KAFKA_VERSION +++ echo 2.4.1 ++ KAFKA_VERSION=2.4.1 + WORK_DIR=/tmp/tidb_cdc_test/storage_cleanup + CDC_BINARY=cdc.test + SINK_TYPE=mysql + EXIST_FILES=() + CLEANED_FILES=() + trap stop_tidb_cluster EXIT + run mysql + '[' mysql '!=' storage ']' + return + check_logs /tmp/tidb_cdc_test/storage_cleanup ++ date + echo '[Fri Apr 26 19:27:10 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>>' [Fri Apr 26 19:27:10 CST 2024] <<<<<< run test case storage_cleanup success! >>>>>> + stop_tidb_cluster =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/csv_storage_basic/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:27:13 CST 2024] <<<<<< run test case csv_storage_basic success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/csv_storage_multi_tables_ddl/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:27:15 CST 2024] <<<<<< run test case csv_storage_multi_tables_ddl success! >>>>>> =================>> Running test /home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/csv_storage_partition_table/run.sh using Sink-Type: mysql... <<================= [Fri Apr 26 19:27:18 CST 2024] <<<<<< run test case csv_storage_partition_table success! >>>>>> \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2671 0 --:--:-- --:--:-- --:--:-- 2695 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:28:02.070","puller_resolved_ts":"2024-04-26 19:27:56.070","last_synced_ts":"2024-04-26 19:25:45.369","now_ts":"2024-04-26 19:28:02.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:28:02.070","puller_resolved_ts":"2024-04-26' '19:27:56.070","last_synced_ts":"2024-04-26' '19:25:45.369","now_ts":"2024-04-26' '19:28:02.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:28:02.070","puller_resolved_ts":"2024-04-26' '19:27:56.070","last_synced_ts":"2024-04-26' '19:25:45.369","now_ts":"2024-04-26' '19:28:02.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68a79ef8001a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:23484, start at 2024-04-26 19:28:25.071850066 +0800 CST m=+5.107648055 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:30:25.078 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:28:25.072 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:18:25.072 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68a79ef8001a Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:23484, start at 2024-04-26 19:28:25.071850066 +0800 CST m=+5.107648055 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:30:25.078 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:28:25.072 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:18:25.072 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68a79fac0014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:23559, start at 2024-04-26 19:28:25.093194544 +0800 CST m=+5.067755567 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:30:25.100 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:28:25.067 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:18:25.067 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24806.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349132909805569 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349132909805569 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349132909805569 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:28:30 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2483824840.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:28:33 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97568d01-cc88-4106-80b8-d290cc2e49d0 {"id":"97568d01-cc88-4106-80b8-d290cc2e49d0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a29c013f8 97568d01-cc88-4106-80b8-d290cc2e49d0 /tidb/cdc/default/default/upstream/7362136140866351401 {"id":7362136140866351401,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97568d01-cc88-4106-80b8-d290cc2e49d0 {"id":"97568d01-cc88-4106-80b8-d290cc2e49d0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a29c013f8 97568d01-cc88-4106-80b8-d290cc2e49d0 /tidb/cdc/default/default/upstream/7362136140866351401 {"id":7362136140866351401,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/97568d01-cc88-4106-80b8-d290cc2e49d0 {"id":"97568d01-cc88-4106-80b8-d290cc2e49d0","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a29c013f8 97568d01-cc88-4106-80b8-d290cc2e49d0 /tidb/cdc/default/default/upstream/7362136140866351401 {"id":7362136140866351401,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349132909805569 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.24887.out cli changefeed create --start-ts=449349132909805569 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362136140866351401,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:28:33.548633179+08:00","start_ts":449349132909805569,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"none","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349132909805569,"checkpoint_ts":449349132909805569,"checkpoint_time":"2024-04-26 19:28:28.622"} PASS coverage: 2.5% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 7457 0 --:--:-- --:--:-- --:--:-- 7531 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:28:28.622","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-04-26 19:28:55.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:28:28.622","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:28:55.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:28:28.622","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:28:55.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_normal_case_and_unavailable_pd conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68aafa4c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:25890, start at 2024-04-26 19:29:20.045384344 +0800 CST m=+5.138943168 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:31:20.055 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:29:20.019 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:19:20.019 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68aafa4c0013 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:25890, start at 2024-04-26 19:29:20.045384344 +0800 CST m=+5.138943168 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:31:20.055 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:29:20.019 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:19:20.019 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68aafd80000d Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:25967, start at 2024-04-26 19:29:20.237325138 +0800 CST m=+5.239562114 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:31:20.244 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:29:20.224 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:19:20.224 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.27231.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349147287355393 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349147287355393 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349147287355393 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:29:24 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + GO_FAILPOINTS= + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2726027262.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + [[ no != \n\o ]] + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:29:27 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6f664aa-b535-4056-b7d5-74bd25740523 {"id":"b6f664aa-b535-4056-b7d5-74bd25740523","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2a94c1f5 b6f664aa-b535-4056-b7d5-74bd25740523 /tidb/cdc/default/default/upstream/7362136376440231384 {"id":7362136376440231384,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6f664aa-b535-4056-b7d5-74bd25740523 {"id":"b6f664aa-b535-4056-b7d5-74bd25740523","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2a94c1f5 b6f664aa-b535-4056-b7d5-74bd25740523 /tidb/cdc/default/default/upstream/7362136376440231384 {"id":7362136376440231384,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/b6f664aa-b535-4056-b7d5-74bd25740523 {"id":"b6f664aa-b535-4056-b7d5-74bd25740523","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2a94c1f5 b6f664aa-b535-4056-b7d5-74bd25740523 /tidb/cdc/default/default/upstream/7362136376440231384 {"id":7362136376440231384,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349147287355393 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.27312.out cli changefeed create --start-ts=449349147287355393 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362136376440231384,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:29:28.390760185+08:00","start_ts":449349147287355393,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349147287355393,"checkpoint_ts":449349147287355393,"checkpoint_time":"2024-04-26 19:29:23.468"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2233 0 --:--:-- --:--:-- --:--:-- 2255 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:29:23.468","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-04-26 19:29:29.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:29:23.468","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:29:29.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:29:23.468","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:29:29.000","info":"Data' syncing is 'finished"}' ++ jq -r .sink_checkpoint_ts + sink_checkpoint_ts='2024-04-26 19:29:23.468' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:29:23.468","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:29:29.000","info":"Data' syncing is 'finished"}' ++ jq -r .puller_resolved_ts + puller_resolved_ts='1970-01-01 08:00:00.000' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:29:23.468","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:29:29.000","info":"Data' syncing is 'finished"}' ++ jq -r .last_synced_ts + last_synced_ts='1970-01-01 08:00:00.000' + '[' true '!=' true ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' + '[' '1970-01-01 08:00:00.000' '!=' '1970-01-01 08:00:00.000' ']' ++ date '+%Y-%m-%d %H:%M:%S' + current='2024-04-26 19:29:30' + echo 'sink_checkpoint_ts is 2024-04-26' 19:29:23.468 sink_checkpoint_ts is 2024-04-26 19:29:23.468 ++ date -d '2024-04-26 19:29:23.468' +%s + checkpoint_timestamp=1714130963 ++ date -d '2024-04-26 19:29:30' +%s + current_timestamp=1714130970 + '[' 7 -gt 300 ']' + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 4086 0 --:--:-- --:--:-- --:--:-- 4118 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:29:34.969","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:29:30.619","now_ts":"2024-04-26 19:29:37.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:29:34.969","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:29:30.619","now_ts":"2024-04-26' '19:29:37.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:29:34.969","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:29:30.619","now_ts":"2024-04-26' '19:29:37.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 2608 0 --:--:-- --:--:-- --:--:-- 2630 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:31:46.118","puller_resolved_ts":"2024-04-26 19:31:40.119","last_synced_ts":"2024-04-26 19:29:30.619","now_ts":"2024-04-26 19:31:47.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:31:46.118","puller_resolved_ts":"2024-04-26' '19:31:40.119","last_synced_ts":"2024-04-26' '19:29:30.619","now_ts":"2024-04-26' '19:31:47.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' + kill_pd ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 25152 7.5 0.0 13912144 142168 ? Sl 19:29 0:12 pd-server --advertise-client-urls http://127.0.0.1:2379 --client-urls http://0.0.0.0:2379 --advertise-peer-urls http://127.0.0.1:2380 --peer-urls http://0.0.0.0:2380 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/pd1.log --data-dir /tmp/tidb_cdc_test/synced_status/pd1 --name=pd1 --initial-cluster=pd1=http://127.0.0.1:2380 jenkins 25216 5.1 0.0 13517264 135612 ? Sl 19:29 0:08 pd-server --advertise-client-urls http://127.0.0.1:2479 --client-urls http://0.0.0.0:2479 --advertise-peer-urls http://127.0.0.1:2480 --peer-urls http://0.0.0.0:2480 --config /tmp/tidb_cdc_test/synced_status/pd-config.toml --log-file /tmp/tidb_cdc_test/synced_status/down_pd.log --data-dir /tmp/tidb_cdc_test/synced_status/down_pd' ++ ps aux ++ grep pd-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 + sleep 20 {"level":"warn","ts":1714131115.078331,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ea7500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714131115.0783908,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714131115.1544082,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ecc700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714131115.1544726,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714131115.2688522,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00386dc00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714131115.268921,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:31:59.962623+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001cca80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:31:59.963053+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001320000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:00.077245+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007fae00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:05.964535+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001320000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:05.964784+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001cca80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:06.077865+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007fae00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:32:11.965858+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001320000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:11.965871+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001cca80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:12.078876+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007fae00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:32:17.966962+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001320000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:17.967676+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001cca80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:18.079823+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007fae00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:32:19.954004+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001320000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:32:19.954051+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:32:19.956923+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001cca80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:32:19.956962+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:32:20.068833+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007fae00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":"2024-04-26T19:32:20.068893+0800","logger":"etcd-client","caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:32:23.968676+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001320000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:23.968924+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001cca80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:24.080874+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007fae00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:32:29.969979+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001320000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:29.970195+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001cca80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":1714131150.0798743,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ea7500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714131150.079939,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":"2024-04-26T19:32:30.081806+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007fae00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"warn","ts":1714131150.155038,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001ecc700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"info","ts":1714131150.1550722,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} {"level":"warn","ts":1714131150.2697976,"caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00386dc00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} {"level":"info","ts":1714131150.2698312,"caller":"v3@v3.5.10/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"} 0 0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0{"level":"warn","ts":"2024-04-26T19:32:35.970783+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001cca80/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:35.97117+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001320000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""} {"level":"warn","ts":"2024-04-26T19:32:36.083235+0800","logger":"etcd-client","caller":"v3@v3.5.10/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0007fae00/127.0.0.1:2479","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2479: connect: connection refused\""} 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 27 100 135 100 135 0 0 4 0 0:00:33 0:00:30 0:00:03 33 + synced_status='{ "error_msg": "[CDC:ErrPDEtcdAPIError]etcd api call error: context deadline exceeded", "error_code": "CDC:ErrPDEtcdAPIError" }' ++ jq -r .error_code ++ echo '{' '"error_msg":' '"[CDC:ErrPDEtcdAPIError]etcd' api call error: context deadline 'exceeded",' '"error_code":' '"CDC:ErrPDEtcdAPIError"' '}' + error_code=CDC:ErrPDEtcdAPIError + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tikv conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b874400009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:28449, start at 2024-04-26 19:33:00.824193204 +0800 CST m=+4.921021084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:35:00.830 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:33:00.816 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:23:00.816 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b874400009 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:28449, start at 2024-04-26 19:33:00.824193204 +0800 CST m=+4.921021084 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:35:00.830 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:33:00.816 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:23:00.816 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68b876680014 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:28535, start at 2024-04-26 19:33:00.992731999 +0800 CST m=+5.032735218 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:35:00.999 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:33:01.004 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:23:01.004 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.29831.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349205233762305 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349205233762305 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349205233762305 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:33:05 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + GO_FAILPOINTS= + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.2986629868.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:33:09 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63dd77e3-1d9f-441a-8177-e6ddff097c82 {"id":"63dd77e3-1d9f-441a-8177-e6ddff097c82","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2df211f2 63dd77e3-1d9f-441a-8177-e6ddff097c82 /tidb/cdc/default/default/upstream/7362137328929947230 {"id":7362137328929947230,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63dd77e3-1d9f-441a-8177-e6ddff097c82 {"id":"63dd77e3-1d9f-441a-8177-e6ddff097c82","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2df211f2 63dd77e3-1d9f-441a-8177-e6ddff097c82 /tidb/cdc/default/default/upstream/7362137328929947230 {"id":7362137328929947230,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/63dd77e3-1d9f-441a-8177-e6ddff097c82 {"id":"63dd77e3-1d9f-441a-8177-e6ddff097c82","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a2df211f2 63dd77e3-1d9f-441a-8177-e6ddff097c82 /tidb/cdc/default/default/upstream/7362137328929947230 {"id":7362137328929947230,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349205233762305 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.29917.out cli changefeed create --start-ts=449349205233762305 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362137328929947230,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:33:09.37552211+08:00","start_ts":449349205233762305,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349205233762305,"checkpoint_ts":449349205233762305,"checkpoint_time":"2024-04-26 19:33:04.516"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tikv ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 27885 23.6 0.4 3822556 1686316 ? Sl 19:32 0:05 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20160 --status-addr 127.0.0.1:20181 --log-file /tmp/tidb_cdc_test/synced_status/tikv1.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv1 jenkins 27886 17.7 0.4 3778012 1634580 ? Sl 19:32 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20161 --status-addr 127.0.0.1:20182 --log-file /tmp/tidb_cdc_test/synced_status/tikv2.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv2 jenkins 27887 17.5 0.4 3775444 1635288 ? Sl 19:32 0:04 tikv-server --pd 127.0.0.1:2379 -A 127.0.0.1:20162 --status-addr 127.0.0.1:20183 --log-file /tmp/tidb_cdc_test/synced_status/tikv3.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv3 jenkins 27889 22.9 0.4 3797464 1676716 ? Sl 19:32 0:05 tikv-server --pd 127.0.0.1:2479 -A 127.0.0.1:21160 --status-addr 127.0.0.1:21180 --log-file /tmp/tidb_cdc_test/synced_status/tikv_down.log --log-level debug -C /tmp/tidb_cdc_test/synced_status/tikv-config.toml -s /tmp/tidb_cdc_test/synced_status/tikv_down' ++ ps aux ++ grep tikv-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2519 0 --:--:-- --:--:-- --:--:-- 2505 100 243 100 243 0 0 2517 0 --:--:-- --:--:-- --:--:-- 2505 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:33:17.016","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:33:11.266","now_ts":"2024-04-26 19:33:18.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:33:17.016","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:33:11.266","now_ts":"2024-04-26' '19:33:18.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:33:17.016","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:33:11.266","now_ts":"2024-04-26' '19:33:18.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 8740 0 --:--:-- --:--:-- --:--:-- 8817 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:33:18.017","puller_resolved_ts":"2024-04-26 19:33:18.017","last_synced_ts":"2024-04-26 19:33:11.266","now_ts":"2024-04-26 19:35:28.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:33:18.017","puller_resolved_ts":"2024-04-26' '19:33:18.017","last_synced_ts":"2024-04-26' '19:33:11.266","now_ts":"2024-04-26' '19:35:28.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:33:18.017","puller_resolved_ts":"2024-04-26' '19:33:18.017","last_synced_ts":"2024-04-26' '19:33:11.266","now_ts":"2024-04-26' '19:35:28.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_unavailable_tidb conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68c2ea40001f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:31055, start at 2024-04-26 19:35:52.256490835 +0800 CST m=+5.059083001 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:37:52.263 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:35:52.258 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:25:52.258 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68c2ea40001f Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:31055, start at 2024-04-26 19:35:52.256490835 +0800 CST m=+5.059083001 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:37:52.263 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:35:52.258 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:25:52.258 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68c2eba80017 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:31135, start at 2024-04-26 19:35:52.332691334 +0800 CST m=+5.087554577 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:37:52.339 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:35:52.298 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:25:52.298 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.32367.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349250228944897 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349250228944897 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349250228944897 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:35:57 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + GO_FAILPOINTS= + [[ no != \n\o ]] + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3239332395.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default + (( i = 0 )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:36:00 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0fd89e26-1303-41fe-aeea-6f13edb9ea58 {"id":"0fd89e26-1303-41fe-aeea-6f13edb9ea58","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a309122f3 0fd89e26-1303-41fe-aeea-6f13edb9ea58 /tidb/cdc/default/default/upstream/7362138067398246659 {"id":7362138067398246659,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0fd89e26-1303-41fe-aeea-6f13edb9ea58 {"id":"0fd89e26-1303-41fe-aeea-6f13edb9ea58","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a309122f3 0fd89e26-1303-41fe-aeea-6f13edb9ea58 /tidb/cdc/default/default/upstream/7362138067398246659 {"id":7362138067398246659,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/0fd89e26-1303-41fe-aeea-6f13edb9ea58 {"id":"0fd89e26-1303-41fe-aeea-6f13edb9ea58","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a309122f3 0fd89e26-1303-41fe-aeea-6f13edb9ea58 /tidb/cdc/default/default/upstream/7362138067398246659 {"id":7362138067398246659,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349250228944897 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.32442.out cli changefeed create --start-ts=449349250228944897 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362138067398246659,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:36:01.068964112+08:00","start_ts":449349250228944897,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349250228944897,"checkpoint_ts":449349250228944897,"checkpoint_time":"2024-04-26 19:35:56.159"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + run_sql 'USE TEST;Create table t1(a int primary key, b int);insert into t1 values(1,2);insert into t1 values(2,3);' + check_table_exists test.t1 127.0.0.1 3306 table test.t1 not exists for 1-th check, retry later table test.t1 exists + sleep 5 + kill_tidb ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status + info='jenkins 31051 3.0 0.0 2704456 164228 ? Sl 19:35 0:00 tidb-server -P 4000 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714131347188302328.toml --store tikv --path 127.0.0.1:2379 --status=10080 --log-file /tmp/tidb_cdc_test/synced_status/tidb.log jenkins 31055 10.8 0.0 2648460 183088 ? Sl 19:35 0:02 tidb-server -P 4001 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714131347191355107.toml --store tikv --path 127.0.0.1:2379 --status=10081 --log-file /tmp/tidb_cdc_test/synced_status/tidb_other.log jenkins 31135 10.8 0.0 2188236 214220 ? Sl 19:35 0:02 tidb-server -P 3306 -config /tmp/tidb_cdc_test/synced_status/tidb-config-1714131347238696774.toml --store tikv --path 127.0.0.1:2479 --status=20080 --log-file /tmp/tidb_cdc_test/synced_status/tidb_down.log' ++ ps aux ++ grep tidb-server ++ grep /tmp/tidb_cdc_test/synced_status ++ awk '{print $2}' ++ xargs kill -9 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 243 100 243 0 0 2264 0 --:--:-- --:--:-- --:--:-- 2271 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:36:08.258","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"2024-04-26 19:36:02.958","now_ts":"2024-04-26 19:36:10.000","info":"The data syncing is not finished, please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:36:08.258","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:36:02.958","now_ts":"2024-04-26' '19:36:10.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:36:08.258","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"2024-04-26' '19:36:02.958","now_ts":"2024-04-26' '19:36:10.000","info":"The' data syncing is not finished, please 'wait"}' ++ jq -r .info + info='The data syncing is not finished, please wait' + target_message='The data syncing is not finished, please wait' + '[' 'The data syncing is not finished, please wait' '!=' 'The data syncing is not finished, please wait' ']' + sleep 130 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 221 100 221 0 0 3236 0 --:--:-- --:--:-- --:--:-- 3250 + synced_status='{"synced":true,"sink_checkpoint_ts":"2024-04-26 19:38:18.358","puller_resolved_ts":"2024-04-26 19:38:11.358","last_synced_ts":"2024-04-26 19:36:02.958","now_ts":"2024-04-26 19:38:20.000","info":"Data syncing is finished"}' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:38:18.358","puller_resolved_ts":"2024-04-26' '19:38:11.358","last_synced_ts":"2024-04-26' '19:36:02.958","now_ts":"2024-04-26' '19:38:20.000","info":"Data' syncing is 'finished"}' ++ jq .synced + status=true + '[' true '!=' true ']' ++ echo '{"synced":true,"sink_checkpoint_ts":"2024-04-26' '19:38:18.358","puller_resolved_ts":"2024-04-26' '19:38:11.358","last_synced_ts":"2024-04-26' '19:36:02.958","now_ts":"2024-04-26' '19:38:20.000","info":"Data' syncing is 'finished"}' ++ jq -r .info + info='Data syncing is finished' + target_message='Data syncing is finished' + '[' 'Data syncing is finished' '!=' 'Data syncing is finished' ']' + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + run_case_with_failpoint conf/changefeed-redo.toml + rm -rf /tmp/tidb_cdc_test/synced_status + mkdir -p /tmp/tidb_cdc_test/synced_status + start_tidb_cluster --workdir /tmp/tidb_cdc_test/synced_status shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory The 1 times to try to start tidb cluster... shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory chdir: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory start tidb cluster in /tmp/tidb_cdc_test/synced_status Starting Upstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Starting Downstream PD... Release Version: v7.5.1-5-g584533652 Edition: Community Git Commit Hash: 58453365285465cd90bc4472cff2bad7ce4d764b Git Branch: release-7.5 UTC Build Time: 2024-04-03 10:04:14 Verifying upstream PD is started... Verifying downstream PD is started... Starting Upstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Downstream TiKV... TiKV Release Version: 7.5.2 Edition: Community Git Commit Hash: 46c2b1e4fa7bc9a71e9294770766c2f2b8bb8990 Git Commit Branch: release-7.5 UTC Build Time: 2024-04-25 09:30:46 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Profile: dist_release Starting Upstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Starting Downstream TiDB... Release Version: v7.5.1-43-ge406d5780b Edition: Community Git Commit Hash: e406d5780b18a1f2aaf6230cde4b6403991e228d Git Branch: release-7.5 UTC Build Time: 2024-04-22 09:56:10 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore Verifying Upstream TiDB is started... ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68cd63080016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:33471, start at 2024-04-26 19:38:43.822846718 +0800 CST m=+4.958192931 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:40:43.829 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:38:43.828 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:28:43.828 +0800 All versions after safe point can be accessed. (DO NOT EDIT) VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68cd63080016 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:33471, start at 2024-04-26 19:38:43.822846718 +0800 CST m=+4.958192931 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:40:43.829 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:38:43.828 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:28:43.828 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Verifying Downstream TiDB is started... VARIABLE_NAME VARIABLE_VALUE COMMENT bootstrapped True Bootstrap flag. Do not delete. tidb_server_version 179 Bootstrap version. Do not delete. system_tz Asia/Shanghai TiDB Global System Timezone. new_collation_enabled True If the new collations are enabled. Do not edit it. ddl_table_version 3 DDL Table Version. Do not delete. tikv_gc_leader_uuid 63c68cd660c0001 Current GC worker leader UUID. (DO NOT EDIT) tikv_gc_leader_desc host:ap-tiflow-release-7-5-pull-cdc-integration-mysql-test-303-cjzhk, pid:33550, start at 2024-04-26 19:38:43.971300083 +0800 CST m=+5.054426024 Host name and pid of current GC leader. (DO NOT EDIT) tikv_gc_leader_lease 20240426-19:40:43.977 +0800 Current GC worker leader lease. (DO NOT EDIT) tikv_gc_auto_concurrency true Let TiDB pick the concurrency automatically. If set false, tikv_gc_concurrency will be used tikv_gc_enable true Current GC enable status tikv_gc_run_interval 10m0s GC run interval, at least 10m, in Go format. tikv_gc_life_time 10m0s All versions within life time will not be collected by GC, at least 10m, in Go format. tikv_gc_last_run_time 20240426-19:38:43.971 +0800 The time when last GC starts. (DO NOT EDIT) tikv_gc_safe_point 20240426-19:28:43.971 +0800 All versions after safe point can be accessed. (DO NOT EDIT) Starting Upstream TiFlash... TiFlash Release Version: v7.5.1-12-g9002cc34d Edition: Community Git Commit Hash: 9002cc34d3b593a718b6c5260ba18f30a45ab314 Git Branch: HEAD UTC Build Time: 2024-04-18 07:24:48 Enable Features: jemalloc sm4(GmSSL) avx2 avx512 unwind thinlto Profile: RELWITHDEBINFO Raft Proxy Git Commit Hash: 521fd9dbc55e58646045d88f91c3c35db50b5981 Git Commit Branch: HEAD UTC Build Time: 2024-04-18 07:28:40 Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14) Storage Engine: tiflash Prometheus Prefix: tiflash_proxy_ Profile: release Enable Features: portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure Verifying Upstream TiFlash is started... Logging trace to /tmp/tidb_cdc_test/synced_status/tiflash/log/server.log Logging errors to /tmp/tidb_cdc_test/synced_status/tiflash/log/error.log arg matches is ArgMatches { args: {"advertise-addr": MatchedArg { occurs: 1, indices: [4], vals: ["127.0.0.1:9000"] }, "engine-addr": MatchedArg { occurs: 1, indices: [2], vals: ["127.0.0.1:9500"] }, "config": MatchedArg { occurs: 1, indices: [8], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash-proxy.toml"] }, "engine-version": MatchedArg { occurs: 1, indices: [12], vals: ["v7.5.1-12-g9002cc34d"] }, "log-file": MatchedArg { occurs: 1, indices: [18], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/log/proxy.log"] }, "engine-label": MatchedArg { occurs: 1, indices: [14], vals: ["tiflash"] }, "pd-endpoints": MatchedArg { occurs: 1, indices: [16], vals: ["127.0.0.1:2379"] }, "data-dir": MatchedArg { occurs: 1, indices: [6], vals: ["/tmp/tidb_cdc_test/synced_status/tiflash/db/proxy"] }, "engine-git-hash": MatchedArg { occurs: 1, indices: [10], vals: ["9002cc34d3b593a718b6c5260ba18f30a45ab314"] }, "addr": MatchedArg { occurs: 1, indices: [20], vals: ["127.0.0.1:9000"] }}, subcommand: None, usage: Some("USAGE:\n TiFlash Proxy [FLAGS] [OPTIONS] --engine-git-hash --engine-label --engine-version ") } + cd /tmp/tidb_cdc_test/synced_status + export 'GO_FAILPOINTS=github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' ++ run_cdc_cli_tso_query 127.0.0.1 2379 + pd_host=127.0.0.1 + pd_port=2379 ++ run_cdc_cli tso query --pd=http://127.0.0.1:2379 + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.34906.out cli tso query --pd=http://127.0.0.1:2379 + set +x + tso='449349295126085637 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/...' + echo 449349295126085637 PASS coverage: 1.9% of statements in github.com/pingcap/tiflow/... + awk -F ' ' '{print $1}' + set +x + start_ts=449349295126085637 + run_cdc_server --workdir /tmp/tidb_cdc_test/synced_status --binary cdc.test [Fri Apr 26 19:38:48 CST 2024] <<<<<< START cdc server in synced_status case >>>>>> + [[ '' == \t\r\u\e ]] + set +e + get_info_fail_msg='failed to get info:' + etcd_info_msg='etcd info' + '[' -z '' ']' + curl_status_cmd='curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info' + [[ no != \n\o ]] + GO_FAILPOINTS='github.com/pingcap/tiflow/cdc/owner/ChangefeedOwnerNotUpdateCheckpoint=return(true)' + (( i = 0 )) + (( i <= 50 )) + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.3494234944.out server --log-file /tmp/tidb_cdc_test/synced_status/cdc.log --log-level debug --data-dir /tmp/tidb_cdc_test/synced_status/cdc_data --cluster-id default ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connection refused * Failed connect to 127.0.0.1:8300; Connection refused * Closing connection 0 + res= + echo '' + grep -q 'failed to get info:' + echo '' + grep -q 'etcd info' + '[' 0 -eq 50 ']' + sleep 3 + (( i++ )) + (( i <= 50 )) ++ curl -vsL --max-time 20 http://127.0.0.1:8300/debug/info * About to connect() to 127.0.0.1 port 8300 (#0) * Trying 127.0.0.1... * Connected to 127.0.0.1 (127.0.0.1) port 8300 (#0) > GET /debug/info HTTP/1.1 > User-Agent: curl/7.29.0 > Host: 127.0.0.1:8300 > Accept: */* > < HTTP/1.1 200 OK < Date: Fri, 26 Apr 2024 11:38:51 GMT < Content-Length: 613 < Content-Type: text/plain; charset=utf-8 < { [data not shown] * Connection #0 to host 127.0.0.1 left intact + res=' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e7e40f6-682f-45c0-8287-6bb53ca4c527 {"id":"7e7e40f6-682f-45c0-8287-6bb53ca4c527","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a332dc3ef 7e7e40f6-682f-45c0-8287-6bb53ca4c527 /tidb/cdc/default/default/upstream/7362138799426581869 {"id":7362138799426581869,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e7e40f6-682f-45c0-8287-6bb53ca4c527 {"id":"7e7e40f6-682f-45c0-8287-6bb53ca4c527","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a332dc3ef 7e7e40f6-682f-45c0-8287-6bb53ca4c527 /tidb/cdc/default/default/upstream/7362138799426581869 {"id":7362138799426581869,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'failed to get info:' + echo ' *** owner info ***: *** processors info ***: *** etcd info ***: /tidb/cdc/default/__cdc_meta__/capture/7e7e40f6-682f-45c0-8287-6bb53ca4c527 {"id":"7e7e40f6-682f-45c0-8287-6bb53ca4c527","address":"127.0.0.1:8300","version":"v7.5.1-57-g10f68e69e"} /tidb/cdc/default/__cdc_meta__/meta/meta-version 1 /tidb/cdc/default/__cdc_meta__/owner/22318f1a332dc3ef 7e7e40f6-682f-45c0-8287-6bb53ca4c527 /tidb/cdc/default/default/upstream/7362138799426581869 {"id":7362138799426581869,"pd-endpoints":"http://127.0.0.1:2379,http://127.0.0.1:2379","key-path":"","cert-path":"","ca-path":"","cert-allowed-cn":null}' + grep -q 'etcd info' + break + set +x + config_path=conf/changefeed-redo.toml + SINK_URI='mysql://root@127.0.0.1:3306/?max-txn-row=1' + run_cdc_cli changefeed create --start-ts=449349295126085637 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml + cdc.test -test.coverprofile=/tmp/tidb_cdc_test/cov.synced_status.cli.35002.out cli changefeed create --start-ts=449349295126085637 '--sink-uri=mysql://root@127.0.0.1:3306/?max-txn-row=1' --changefeed-id=test-1 --config=/home/jenkins/agent/workspace/pingcap/tiflow/release-7.5/pull_cdc_integration_mysql_test/tiflow/tests/integration_tests/synced_status/conf/changefeed-redo.toml Create changefeed successfully! ID: test-1 Info: {"upstream_id":7362138799426581869,"namespace":"default","id":"test-1","sink_uri":"mysql://root@127.0.0.1:3306/?max-txn-row=1","create_time":"2024-04-26T19:38:52.26975002+08:00","start_ts":449349295126085637,"config":{"memory_quota":1073741824,"case_sensitive":false,"force_replicate":false,"ignore_ineligible_table":false,"check_gc_safe_point":true,"enable_sync_point":false,"bdr_mode":false,"sync_point_interval":600000000000,"sync_point_retention":86400000000000,"filter":{"rules":["*.*"]},"mounter":{"worker_num":16},"sink":{"csv":{"delimiter":",","quote":"\"","null":"\\N","include_commit_ts":false,"binary_encoding_method":"base64"},"encoder_concurrency":32,"terminator":"\r\n","date_separator":"day","enable_partition_separator":true,"enable_kafka_sink_v2":false,"only_output_updated_columns":false,"delete_only_output_handle_key_columns":false,"advance_timeout":150},"consistent":{"level":"eventual","max_log_size":64,"flush_interval":2000,"meta_flush_interval":200,"encoding_worker_num":16,"flush_worker_num":8,"storage":"file:///tmp/tidb_cdc_test/synced_status/redo","use_file_backend":false,"memory_usage":{"memory_quota_percentage":50,"event_cache_percentage":0}},"scheduler":{"enable_table_across_nodes":false,"region_threshold":100000,"write_key_threshold":0},"integrity":{"integrity_check_level":"none","corruption_handle_level":"warn"},"changefeed_error_stuck_duration":1800000000000,"sql_mode":"ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION","synced_status":{"synced_check_interval":120,"checkpoint_interval":20}},"state":"normal","creator_version":"v7.5.1-57-g10f68e69e","resolved_ts":449349295126085637,"checkpoint_ts":449349295126085637,"checkpoint_time":"2024-04-26 19:38:47.428"} PASS coverage: 2.6% of statements in github.com/pingcap/tiflow/... + set +x + sleep 20 ++ curl -X GET http://127.0.0.1:8300/api/v2/changefeeds/test-1/synced % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 723 100 723 0 0 11739 0 --:--:-- --:--:-- --:--:-- 11852 + synced_status='{"synced":false,"sink_checkpoint_ts":"2024-04-26 19:38:47.428","puller_resolved_ts":"1970-01-01 08:00:00.000","last_synced_ts":"1970-01-01 08:00:00.000","now_ts":"2024-04-26 19:39:13.000","info":"Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' \u003e '\''Resolved-Ts'\'' \u003e '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait"}' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:38:47.428","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:39:13.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq .synced + status=false + '[' false '!=' false ']' ++ echo '{"synced":false,"sink_checkpoint_ts":"2024-04-26' '19:38:47.428","puller_resolved_ts":"1970-01-01' '08:00:00.000","last_synced_ts":"1970-01-01' '08:00:00.000","now_ts":"2024-04-26' '19:39:13.000","info":"Please' check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view ''\''TiKV-Details'\''' '\u003e' ''\''Resolved-Ts'\''' '\u003e' ''\''Max' Leader Resolved TS 'gap'\''' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please 'wait"}' ++ jq -r .info + info='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + target_message='Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' + '[' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' '!=' 'Please check whether PD is online and TiKV Regions are all available. If PD is offline or some TiKV regions are not available, it means that the data syncing process is complete. To check whether TiKV regions are all available, you can view '\''TiKV-Details'\'' > '\''Resolved-Ts'\'' > '\''Max Leader Resolved TS gap'\'' on Grafana. If the gap is large, such as a few minutes, it means that some regions in TiKV are unavailable. Otherwise, if the gap is small and PD is online, it means the data syncing is incomplete, so please wait' ']' + export GO_FAILPOINTS= + GO_FAILPOINTS= + cleanup_process cdc.test wait process cdc.test exit for 1-th time... wait process cdc.test exit for 2-th time... cdc.test: no process found wait process cdc.test exit for 3-th time... process cdc.test already exit + stop_tidb_cluster + check_logs /tmp/tidb_cdc_test/synced_status ++ date + echo '[Fri Apr 26 19:39:24 CST 2024] <<<<<< run test case synced_status success! >>>>>>' [Fri Apr 26 19:39:24 CST 2024] <<<<<< run test case synced_status success! >>>>>> + stop_tidb_cluster \033[0;36m<<< Run all test success >>>\033[0m [Pipeline] } Cache not saved (ws/jenkins-pingcap-tiflow-release-7.5-pull_cdc_integration_mysql_test-303/tiflow-cdc already exists) [Pipeline] // cache [Pipeline] } [Pipeline] // dir [Pipeline] } [Pipeline] // withCredentials [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // parallel [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // timeout [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // container [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] } [Pipeline] // podTemplate [Pipeline] End of Pipeline Finished: SUCCESS